ClusterLogDestination
Scope: Cluster
Version: v1alpha1
Describes setting for a log storage, which you can use in many log sources.
metadata.name
— is an upstream name, which you should use in CustomResource ClusterLoggingConfig.
- specobject
Required value
- spec.elasticsearchobject
- spec.elasticsearch.authobject
- spec.elasticsearch.auth.awsAccessKeystring
Base64-encoded AWS
ACCESS_KEY
. - spec.elasticsearch.auth.awsAssumeRolestring
The ARN of an IAM role to assume at startup.
- spec.elasticsearch.auth.awsRegionstring
AWS region for authentication.
- spec.elasticsearch.auth.awsSecretKeystring
Base64-encoded AWS
SECRET_KEY
. - spec.elasticsearch.auth.passwordstring
Base64-encoded Basic authentication password.
- spec.elasticsearch.auth.strategystring
The authentication strategy to use.
Default:
"Basic"
Allowed values:
Basic
,AWS
- spec.elasticsearch.auth.userstring
The Basic authentication user name.
- spec.elasticsearch.auth.awsAccessKeystring
- spec.elasticsearch.dataStreamEnabledboolean
Use for storage indexes or datastreams (https://www.elastic.co/guide/en/elasticsearch/reference/master/data-streams.html).
Datastream usage is better for logs and metrics storage but they works only for Elasticsearch >= 7.16.X.
Default:
false
- spec.elasticsearch.docTypestring
The
doc_type
for your index data. This is only relevant for Elasticsearch <= 6.X.- For Elasticsearch >= 7.X you do not need this option since this version has removed
doc_type
mapping; - For Elasticsearch >= 6.X the recommended value is
_doc
, because using it will make it easy to upgrade to 7.X; - For Elasticsearch < 6.X you can’t use a value starting with
_
or empty string. Use, for example, values likelogs
.
- For Elasticsearch >= 7.X you do not need this option since this version has removed
- spec.elasticsearch.endpointstring
Required value
Base URL of the Elasticsearch instance.
- spec.elasticsearch.indexstring
Index name to write events to.
- spec.elasticsearch.pipelinestring
Name of the pipeline to apply.
- spec.elasticsearch.tlsobject
Configures the TLS options for outgoing connections.
- spec.elasticsearch.tls.caFilestring
Base64-encoded CA certificate in PEM format.
- spec.elasticsearch.tls.clientCrtobject
Configures the client certificate for outgoing connections.
- spec.elasticsearch.tls.clientCrt.crtFilestring
Required value
Base64-encoded certificate in PEM format.
You must also set the
keyFile
parameter. - spec.elasticsearch.tls.clientCrt.keyFilestring
Required value
Base64-encoded private key in PEM format (PKCS#8).
You must also set the
crtFile
parameter. - spec.elasticsearch.tls.clientCrt.keyPassstring
Base64-encoded pass phrase used to unlock the encrypted key file.
- spec.elasticsearch.tls.clientCrt.crtFilestring
- spec.elasticsearch.tls.verifyCertificateboolean
Validate the TLS certificate of the remote host. Specifically the issuer is checked but not CRLs (Certificate Revocation Lists).
Default:
true
- spec.elasticsearch.tls.verifyHostnameboolean
Verifies that the name of the remote host matches the name specified in the remote host’s TLS certificate.
Default:
true
- spec.elasticsearch.tls.caFilestring
- spec.elasticsearch.authobject
- spec.extraLabelsobject
A set of labels that will be attached to each batch of events.
You can use simple templating here:
{{ app }}
.There are some reserved keys:
- parsed_data
- pod
- pod_labels_*
- pod_ip
- namespace
- image
- container
- node
- pod_owner
Example:
extraLabels: forwarder: vector key: value app_info: "{{ app }}" array_member: "{{ array[0] }}" symbol_escating_value: "{{ pay\.day }}"
- spec.kafkaobject
- spec.kafka.bootstrapServersarray of strings
Required value
A list of host and port pairs that are the addresses of the Kafka brokers in a “bootstrap” Kafka cluster that a Kafka client connects to initially to bootstrap itself.
Default:
[]
- Element of the arraystring
Pattern:
^(.+)\:\d{1,5}$
- Element of the arraystring
- spec.kafka.tlsobject
Configures the TLS options for outgoing connections.
- spec.kafka.tls.caFilestring
Base64-encoded CA certificate in PEM format.
- spec.kafka.tls.clientCrtobject
Configures the client certificate for outgoing connections.
- spec.kafka.tls.clientCrt.crtFilestring
Required value
Base64-encoded certificate in PEM format.
You must also set the
keyFile
parameter. - spec.kafka.tls.clientCrt.keyFilestring
Required value
Base64-encoded private key in PEM format (PKCS#8).
You must also set the
crtFile
parameter. - spec.kafka.tls.clientCrt.keyPassstring
Base64-encoded pass phrase used to unlock the encrypted key file.
- spec.kafka.tls.clientCrt.crtFilestring
- spec.kafka.tls.verifyCertificateboolean
Validate the TLS certificate of the remote host.
Default:
true
- spec.kafka.tls.verifyHostnameboolean
Verifies that the name of the remote host matches the name specified in the remote host’s TLS certificate.
Default:
true
- spec.kafka.tls.caFilestring
- spec.kafka.topicstring
Required value
The Kafka topic name to write events to. This parameter supports template syntax, which enables you to use dynamic per-event values.
- spec.kafka.bootstrapServersarray of strings
- spec.logstashobject
- spec.logstash.endpointstring
Required value
Base URL of the Logstash instance.
- spec.logstash.tlsobject
Configures the TLS options for outgoing connections.
- spec.logstash.tls.caFilestring
Base64-encoded CA certificate in PEM format.
- spec.logstash.tls.clientCrtobject
Configures the client certificate for outgoing connections.
- spec.logstash.tls.clientCrt.crtFilestring
Required value
Base64-encoded certificate in PEM format.
You must also set the
keyFile
parameter. - spec.logstash.tls.clientCrt.keyFilestring
Required value
Base64-encoded private key in PEM format (PKCS#8).
You must also set the
crtFile
parameter. - spec.logstash.tls.clientCrt.keyPassstring
Base64-encoded pass phrase used to unlock the encrypted key file.
- spec.logstash.tls.clientCrt.crtFilestring
- spec.logstash.tls.verifyCertificateboolean
Validate the TLS certificate of the remote host.
Default:
true
- spec.logstash.tls.verifyHostnameboolean
Verifies that the name of the remote host matches the name specified in the remote host’s TLS certificate.
Default:
true
- spec.logstash.tls.caFilestring
- spec.logstash.endpointstring
- spec.lokiobject
- spec.loki.authobject
- spec.loki.auth.passwordstring
Base64-encoded Basic authentication password.
- spec.loki.auth.strategystring
The authentication strategy to use.
Default:
"Basic"
Allowed values:
Basic
,Bearer
- spec.loki.auth.tokenstring
The token to use for Bearer authentication.
- spec.loki.auth.userstring
The Basic authentication user name.
- spec.loki.auth.passwordstring
- spec.loki.endpointstring
Required value
Base URL of the Loki instance.
Agent automatically adds
/loki/api/v1/push
into URL during data transmission. - spec.loki.tlsobject
Configures the TLS options for outgoing connections.
- spec.loki.tls.caFilestring
Base64-encoded CA certificate in PEM format.
- spec.loki.tls.clientCrtobject
Configures the client certificate for outgoing connections.
- spec.loki.tls.clientCrt.crtFilestring
Required value
Base64-encoded certificate in PEM format.
You must also set the
keyFile
parameter. - spec.loki.tls.clientCrt.keyFilestring
Required value
Base64-encoded private key in PEM format (PKCS#8).
You must also set the
crtFile
parameter. - spec.loki.tls.clientCrt.keyPassstring
Base64-encoded pass phrase used to unlock the encrypted key file.
- spec.loki.tls.clientCrt.crtFilestring
- spec.loki.tls.verifyHostnameboolean
Verifies that the name of the remote host matches the name specified in the remote host’s TLS certificate.
Default:
true
- spec.loki.tls.caFilestring
- spec.loki.authobject
- spec.rateLimitobject
Parameter for limiting the flow of events.
- spec.rateLimit.linesPerMinutenumber
Required value
The number of records per minute.
- spec.rateLimit.linesPerMinutenumber
- spec.splunkobject
- spec.splunk.endpointstring
Required value
Base URL of the Splunk instance.
Example:
endpoint: https://http-inputs-hec.splunkcloud.com
- spec.splunk.indexstring
Index name to write events to.
- spec.splunk.tlsobject
Configures the TLS options for outgoing connections.
- spec.splunk.tls.caFilestring
Base64-encoded CA certificate in PEM format.
- spec.splunk.tls.clientCrtobject
Configures the client certificate for outgoing connections.
- spec.splunk.tls.clientCrt.crtFilestring
Required value
Base64-encoded certificate in PEM format.
You must also set the
keyFile
parameter. - spec.splunk.tls.clientCrt.keyFilestring
Required value
Base64-encoded private key in PEM format (PKCS#8).
You must also set the
crtFile
parameter. - spec.splunk.tls.clientCrt.keyPassstring
Base64-encoded pass phrase used to unlock the encrypted key file.
- spec.splunk.tls.clientCrt.crtFilestring
- spec.splunk.tls.verifyCertificateboolean
Validate the TLS certificate of the remote host.
Default:
true
- spec.splunk.tls.verifyHostnameboolean
Verifies that the name of the remote host matches the name specified in the remote host’s TLS certificate.
Default:
true
- spec.splunk.tls.caFilestring
- spec.splunk.tokenstring
Required value
Default Splunk HEC token. If an event has a token set in its metadata, it will have priority over the one set here.
- spec.splunk.endpointstring
- spec.typestring
Type of a log storage backend.
Allowed values:
Loki
,Elasticsearch
,Logstash
,Vector
,Kafka
,Splunk
- spec.vectorobject
- spec.vector.endpointstring
Required value
An address of the Vector instance. API v2 must be used for communication between instances.
Pattern:
^(.+):([0-9]{1,5})$
- spec.vector.tlsobject
Configures the TLS options for outgoing connections.
- spec.vector.tls.caFilestring
Base64-encoded CA certificate in PEM format.
- spec.vector.tls.clientCrtobject
Configures the client certificate for outgoing connections.
- spec.vector.tls.clientCrt.crtFilestring
Required value
Base64-encoded certificate in PEM format.
You must also set the
keyFile
parameter. - spec.vector.tls.clientCrt.keyFilestring
Required value
Base64-encoded private key in PEM format (PKCS#8).
You must also set the
crtFile
parameter. - spec.vector.tls.clientCrt.keyPassstring
Base64-encoded passphrase used to unlock the encrypted key file.
- spec.vector.tls.clientCrt.crtFilestring
- spec.vector.tls.verifyCertificateboolean
Validate the TLS certificate of the remote host.
Default:
true
- spec.vector.tls.verifyHostnameboolean
Verifies that the name of the remote host matches the name specified in the remote host’s TLS certificate.
Default:
true
- spec.vector.tls.caFilestring
- spec.vector.endpointstring
- spec.elasticsearchobject
ClusterLoggingConfig
Scope: Cluster
Version: v1alpha1
Describes a log source in log-pipeline.
Each CustomResource ClusterLoggingConfig
describes rules for log fetching from cluster.
- specobject
Required value
- spec.destinationRefsarray of strings
Required value
Array of
ClusterLogDestination
CustomResource names which this source will output with.Fields with float or boolean values will be converted to strings.
- spec.fileobject
- spec.file.excludearray of strings
Array of file patterns to exclude.
Example:
exclude: - "/var/log/nginx/error.log" - "/var/log/audit.log"
- spec.file.includearray of strings
Array of file patterns to include.
Example:
include: - "/var/log/*.log" - "/var/log/nginx/*.log"
- spec.file.lineDelimiterstring
String sequence used to separate one file line from another.
Example:
lineDelimiter: "\\r\\n"
- spec.file.excludearray of strings
- spec.kubernetesPodsobject
- spec.kubernetesPods.labelSelectorobject
Specifies the label selector to filter Pods with.
You can get more into here.
- spec.kubernetesPods.labelSelector.matchExpressionsarray of objects
List of label expressions for Pods.
Example:
matchExpressions: - key: tier operator: In values: - production - staging
- spec.kubernetesPods.labelSelector.matchExpressions.keystring
Required value
- spec.kubernetesPods.labelSelector.matchExpressions.operatorstring
Required value
Allowed values:
In
,NotIn
,Exists
,DoesNotExist
- spec.kubernetesPods.labelSelector.matchExpressions.valuesarray of strings
- spec.kubernetesPods.labelSelector.matchExpressions.keystring
- spec.kubernetesPods.labelSelector.matchLabelsobject
List of labels which Pod should have.
Example:
matchLabels: foo: bar baz: who
- spec.kubernetesPods.labelSelector.matchExpressionsarray of objects
- spec.kubernetesPods.namespaceSelectorobject
Specifies the Namespace selector to filter Pods with.
- spec.kubernetesPods.namespaceSelector.excludeNamesarray of strings
Include all namespaces except a particular set.
- spec.kubernetesPods.namespaceSelector.labelSelectorobject
Specifies the label selector to filter namespaces.
You can get more into here.
- spec.kubernetesPods.namespaceSelector.labelSelector.matchExpressionsarray of objects
List of label expressions for namespaces.
Example:
matchExpressions: - key: tier operator: In values: - production - staging
- spec.kubernetesPods.namespaceSelector.labelSelector.matchExpressions.keystring
Required value
- spec.kubernetesPods.namespaceSelector.labelSelector.matchExpressions.operatorstring
Required value
Allowed values:
In
,NotIn
,Exists
,DoesNotExist
- spec.kubernetesPods.namespaceSelector.labelSelector.matchExpressions.valuesarray of strings
- spec.kubernetesPods.namespaceSelector.labelSelector.matchExpressions.keystring
- spec.kubernetesPods.namespaceSelector.labelSelector.matchLabelsobject
List of labels which a namespace should have.
Example:
matchLabels: foo: bar baz: who
- spec.kubernetesPods.namespaceSelector.labelSelector.matchExpressionsarray of objects
- spec.kubernetesPods.namespaceSelector.matchNamesarray of strings
Include only a particular set of namespaces.
- spec.kubernetesPods.namespaceSelector.excludeNamesarray of strings
- spec.kubernetesPods.labelSelectorobject
- spec.labelFilterarray of objects
Rules to filter log lines by their labels.
Example:
labelFilter: - field: container operator: In values: - nginx - field: pod_labels.tier operator: Regex values: - prod-.+ - stage-.+
- spec.labelFilter.fieldstring
Required value
Label name for filtering. Must not be empty.
Pattern:
.+
- spec.labelFilter.operatorstring
Required value
Operator for log field comparations:
In
— finds a substring in a string.NotIn
— is a negative version of theIn
operator.Regex
— is trying to match regexp over the field; only log events with matching fields will pass.NotRegex
— is a negative version of theRegex
operator; log events without fields or with not matched fields will pass.Exists
— drops log event if it contains some fields.DoesNotExist
— drops log event if it does not contain some fields.
Allowed values:
In
,NotIn
,Regex
,NotRegex
,Exists
,DoesNotExist
- spec.labelFilter.valuesarray
Array of values or regexes for corresponding operations. Does not work for
Exists
andDoesNotExist
operations.Fields a with float or boolean values will be converted to strings during comparison.
- spec.labelFilter.fieldstring
- spec.logFilterarray of objects
List of filter for logs.
Only matched lines would be stored to log destination.
Example:
logFilter: - field: tier operator: Exists - field: foo operator: NotIn values: - dev - 42 - "true" - "3.14" - field: bar operator: Regex values: - ^abc - ^\d.+$
- spec.logFilter.fieldstring
Required value
Field name for filtering. It should be empty for non-JSON messages.
- spec.logFilter.operatorstring
Required value
Operator for log field comparations:
In
— finds a substring in a string.NotIn
— is a negative version of theIn
operator.Regex
— is trying to match regexp over the field; only log events with matching fields will pass.NotRegex
— is a negative version of theRegex
operator; log events without fields or with not matched fields will pass.Exists
— drops log event if it contains some fields.DoesNotExist
— drops log event if it does not contain some fields.
Allowed values:
In
,NotIn
,Regex
,NotRegex
,Exists
,DoesNotExist
- spec.logFilter.valuesarray
Array of values or regexes for corresponding operations. Does not work for
Exists
andDoesNotExist
operations.Fields a with float or boolean values will be converted to strings during comparison.
- spec.logFilter.fieldstring
- spec.multilineParserobject
Multiline parser for different patterns
- spec.multilineParser.typestring
Required value
Parser types:
None
— do not parse logs.General
— tries to match general multiline logs with space or tabulation on extra lines.Backslash
— tries to match bash style logs with backslash on all lines except the last event line.LogWithTime
— tries to detect events by timestamp.MultilineJSON
— tries to match JSON logs, assuming the event starts with the{
symbol.
Default:
"None"
Allowed values:
None
,General
,Backslash
,LogWithTime
,MultilineJSON
- spec.multilineParser.typestring
- spec.typestring
Required value
Set on of possible input sources.
KubernetesPods
source reads logs from Kubernetes Pods.File
source reads local file from node filesystem.Allowed values:
KubernetesPods
,File
- spec.destinationRefsarray of strings
PodLoggingConfig
Scope: Namespaced
Version: v1alpha1
CustomResource for namespaced Kubernetes source.
Each CustomResource PodLoggingConfig
describes rules for log fetching from specified Namespace.
- specobject
Required value
- spec.clusterDestinationRefsarray of strings
Required value
Array of
ClusterLogDestination
CustomResource names which this source will output with. - spec.labelFilterarray of objects
Rules to filter log lines by their labels.
Example:
labelFilter: - field: container operator: In values: - nginx - field: pod_labels.tier operator: Regex values: - prod-.+ - stage-.+
- spec.labelFilter.fieldstring
Required value
Label name for filtering. Must not be empty.
Pattern:
.+
- spec.labelFilter.operatorstring
Required value
Operator for log field comparations:
In
— finds a substring in a string.NotIn
— is a negative version of theIn
operator.Regex
— is trying to match regexp over the field; only log events with matching fields will pass.NotRegex
— is a negative version of theRegex
operator; log events without fields or with not matched fields will pass.Exists
— drops log event if it contains some fields.DoesNotExist
— drops log event if it does not contain some fields.
Allowed values:
In
,NotIn
,Regex
,NotRegex
,Exists
,DoesNotExist
- spec.labelFilter.valuesarray
Array of values or regexes for corresponding operations. Does not work for
Exists
andDoesNotExist
operations.Fields with a float or boolean values will be converted to strings during comparison.
- spec.labelFilter.fieldstring
- spec.labelSelectorobject
Specifies the label selector to filter Pods.
You can get more into here.
- spec.labelSelector.matchExpressionsarray of objects
List of label expressions for Pods.
Example:
matchExpressions: - key: tier operator: In values: - production - staging
- spec.labelSelector.matchExpressions.keystring
Required value
- spec.labelSelector.matchExpressions.operatorstring
Required value
Allowed values:
In
,NotIn
,Exists
,DoesNotExist
- spec.labelSelector.matchExpressions.valuesarray of strings
- spec.labelSelector.matchExpressions.keystring
- spec.labelSelector.matchLabelsobject
List of labels which Pod should have.
Example:
matchLabels: foo: bar baz: who
- spec.labelSelector.matchExpressionsarray of objects
- spec.logFilterarray of objects
List of filter for logs.
Only matched lines would be stored to log destination.
Example:
logFilter: - field: tier operator: Exists - field: foo operator: NotIn values: - dev - 42 - "true" - "3.14" - field: bar operator: Regex values: - ^abc - ^\d.+$
- spec.logFilter.fieldstring
Required value
Field name for filtering. It should be empty for non-JSON messages.
- spec.logFilter.operatorstring
Required value
Operator for log field comparations:
In
— finds a substring in a string.NotIn
— is a negative version of theIn
operator.Regex
— is trying to match regexp over the field; only log events with matching fields will pass.NotRegex
— is a negative version of theRegex
operator; log events without fields or with not matched fields will pass.Exists
— drops log event if it contains some fields.DoesNotExist
— drops log event if it does not contain some fields.
Allowed values:
In
,NotIn
,Regex
,NotRegex
,Exists
,DoesNotExist
- spec.logFilter.valuesarray
Array of values or regexes for corresponding operations. Does not work for
Exists
andDoesNotExist
operations.Fields a with float or boolean values will be converted to strings during comparison.
- spec.logFilter.fieldstring
- spec.multilineParserobject
Multiline parser for different patterns.
- spec.multilineParser.typestring
Required value
Parser types:
None
— do not parse logs.General
— tries to match general multiline logs with space or tabulation on extra lines.Backslash
— tries to match bash style logs with backslash on all lines except the last event line.LogWithTime
— tries to detect events by timestamp.MultilineJSON
— tries to match JSON logs, assuming the event starts with the{
symbol.
Default:
"None"
Allowed values:
None
,General
,Backslash
,LogWithTime
,MultilineJSON
- spec.multilineParser.typestring
- spec.clusterDestinationRefsarray of strings