The log-shipper module: Custom Resources

Disk buffer parameters.

The type of buffer to use.

Event handling behavior when a buffer is full.

Use for storage indexes or datastreams (https://www.elastic.co/guide/en/elasticsearch/reference/master/data-streams.html).

Datastream usage is better for logs and metrics storage but they works only for Elasticsearch >= 7.16.X.

The doc_type for your index data. This is only relevant for Elasticsearch <= 6.X.

• For Elasticsearch >= 7.X you do not need this option since this version has removed doc_type mapping;
• For Elasticsearch >= 6.X the recommended value is _doc, because using it will make it easy to upgrade to 7.X;
• For Elasticsearch < 6.X you can’t use a value starting with _ or empty string. Use, for example, values like logs.

Base URL of the Elasticsearch instance.

Index name to write events to.

Name of the pipeline to apply.

Configures the TLS options for outgoing connections.

A list of host and port pairs that are the addresses of the Kafka brokers in a “bootstrap” Kafka cluster that a Kafka client connects to initially to bootstrap itself.

How to encode the message.

Allows to set the key_field.

Configuration for SASL authentication when interacting with Kafka.

Configures the TLS options for outgoing connections.

The Kafka topic name to write events to. This parameter supports template syntax, which enables you to use dynamic per-event values.

Base URL of the Logstash instance.

Configures the TLS options for outgoing connections.

Base URL of the Loki instance.

Agent automatically adds /loki/api/v1/push into URL during data transmission.

ID of a tenant.

This option is used only for GrafanaCloud. When running Loki locally, a tenant ID is not required.

Configures the TLS options for outgoing connections.

List of excludes for keyField.

Only NOT matched log entries would be rate limited.

The name of the log field whose value will be hashed to determine if the event should be rate limited.

The number of records per minute.

Address of the socket.

How to encode the message.

Base URL of the Splunk instance.

Index name to write events to.

Configures the TLS options for outgoing connections.

Default Splunk HEC token. If an event has a token set in its metadata, it will have priority over the one set here.

An address of the Vector instance. API v2 must be used for communication between instances.

Configures the TLS options for outgoing connections.

Array of file patterns to exclude when collecting logs.

Wildcards are supported.

Array of file patterns to include.

Wildcards are supported

String sequence used to separate one file line from another.

Specifies the time to keep deleted files opened for reading. Vector will keep pods metadata for this time as well to read logs from deleted pods. This option is useful in cases of a log storage unavailability or a network partition. Vector will keep log files opened until finally sending them to the destination.

Enabling this option may affect the resource consumption of the Vector and also flood a disk with deleted logs. Use it with caution.

The format is a string containing the time unit in hours and minutes: 30m, 1h, 2h30m, 24h.

Specifies the label selector to filter Pods with.

You can get more into here.

Specifies the namespace selector to filter Pods with.

The filter can use one of the three available ways to set the condition (parameters matchNames, excludeNames, labelSelector)

Label name for filtering.

Must not be empty.

Operator for log field comparations:

• In — finds a substring in a string.
• NotIn — is a negative version of the In operator.
• Regex — is trying to match regexp over the field; only log events with matching fields will pass.
• NotRegex — is a negative version of the Regex operator; log events without fields or with not matched fields will pass.
• Exists — drops log event if it contains some fields.
• DoesNotExist — drops log event if it does not contain some fields.

Array of values or regexes for corresponding operations. Does not work for Exists and DoesNotExist operations.

Fields a with float or boolean values will be converted to strings during comparison.

Field name for filtering. It should be empty for non-JSON messages.

Operator for log field comparations:

• In — finds a substring in a string.
• NotIn — is a negative version of the In operator.
• Regex — is trying to match regexp over the field; only log events with matching fields will pass.
• NotRegex — is a negative version of the Regex operator; log events without fields or with not matched fields will pass.
• Exists — drops log event if it contains some fields.
• DoesNotExist — drops log event if it does not contain some fields.

Array of values or regexes for corresponding operations. Does not work for Exists and DoesNotExist operations.

Fields a with float or boolean values will be converted to strings during comparison.

Multiline parser custom regex rules.

Parser types:

• None — do not parse logs.
• General — tries to match general multiline logs with space or tabulation on extra lines.
• Backslash — tries to match bash style logs with backslash on all lines except the last event line.
• LogWithTime — tries to detect events by timestamp.
• MultilineJSON — tries to match JSON logs, assuming the event starts with the { symbol.
• Custom - tries to match logs with the user provided regex in spec.multilineParser.custom field.

Label name for filtering.

Must not be empty.

Operator for log field comparations:

• In — finds a substring in a string.
• NotIn — is a negative version of the In operator.
• Regex — is trying to match regexp over the field; only log events with matching fields will pass.
• NotRegex — is a negative version of the Regex operator; log events without fields or with not matched fields will pass.
• Exists — drops log event if it contains some fields.
• DoesNotExist — drops log event if it does not contain some fields.

Array of values or regexes for corresponding operations. Does not work for Exists and DoesNotExist operations.

Fields with a float or boolean values will be converted to strings during comparison.

List of label expressions for Pods.

List of labels which Pod should have.

Field name for filtering. It should be empty for non-JSON messages.

Operator for log field comparations:

• In — finds a substring in a string.
• NotIn — is a negative version of the In operator.
• Regex — is trying to match regexp over the field; only log events with matching fields will pass.
• NotRegex — is a negative version of the Regex operator; log events without fields or with not matched fields will pass.
• Exists — drops log event if it contains some fields.
• DoesNotExist — drops log event if it does not contain some fields.

Array of values or regexes for corresponding operations. Does not work for Exists and DoesNotExist operations.

Fields a with float or boolean values will be converted to strings during comparison.

Multiline parser custom regex rules.

Parser types:

• None — do not parse logs.
• General — tries to match general multiline logs with space or tabulation on extra lines.
• Backslash — tries to match bash style logs with backslash on all lines except the last event line.
• LogWithTime — tries to detect events by timestamp.
• MultilineJSON — tries to match JSON logs, assuming the event starts with the { symbol.
• Custom - tries to match logs with the user provided regex in spec.multilineParser.custom field.