The log-shipper module: Custom Resources

Required value

• spec.elasticsearchobject
  • spec.elasticsearch.authobject
    • spec.elasticsearch.auth.awsAccessKeystring

      Base64-encoded AWS ACCESS_KEY.

    • spec.elasticsearch.auth.awsAssumeRolestring

      The ARN of an IAM role to assume at startup.

    • spec.elasticsearch.auth.awsRegionstring

      AWS region for authentication.

    • spec.elasticsearch.auth.awsSecretKeystring

      Base64-encoded AWS SECRET_KEY.

    • spec.elasticsearch.auth.passwordstring

      Base64-encoded Basic authentication password.

    • spec.elasticsearch.auth.strategystring

      The authentication strategy to use.

      Default: "Basic"

      Allowed values: Basic, AWS

    • spec.elasticsearch.auth.userstring

      The Basic authentication user name.

  • spec.elasticsearch.dataStreamEnabledboolean

    Use for storage indexes or datastreams (https://www.elastic.co/guide/en/elasticsearch/reference/master/data-streams.html).

    Datastream usage is better for logs and metrics storage but they works only for Elasticsearch >= 7.16.X.

    Default: false

  • spec.elasticsearch.docTypestring

    The doc_type for your index data. This is only relevant for Elasticsearch <= 6.X.

    • For Elasticsearch >= 7.X you do not need this option since this version has removed doc_type mapping;
    • For Elasticsearch >= 6.X the recommended value is _doc, because using it will make it easy to upgrade to 7.X;
    • For Elasticsearch < 6.X you can’t use a value starting with _ or empty string. Use, for example, values like logs.
  • spec.elasticsearch.endpointstring

    Required value

    Base URL of the Elasticsearch instance.

  • spec.elasticsearch.indexstring

    Index name to write events to.

  • spec.elasticsearch.pipelinestring

    Name of the pipeline to apply.

  • spec.elasticsearch.tlsobject

    Configures the TLS options for outgoing connections.

    • spec.elasticsearch.tls.caFilestring

      Base64-encoded CA certificate in PEM format.

    • spec.elasticsearch.tls.clientCrtobject

      Configures the client certificate for outgoing connections.

      • spec.elasticsearch.tls.clientCrt.crtFilestring

        Required value

        Base64-encoded certificate in PEM format.

        You must also set the keyFile parameter.

      • spec.elasticsearch.tls.clientCrt.keyFilestring

        Required value

        Base64-encoded private key in PEM format (PKCS#8).

        You must also set the crtFile parameter.

      • spec.elasticsearch.tls.clientCrt.keyPassstring

        Base64-encoded pass phrase used to unlock the encrypted key file.

    • spec.elasticsearch.tls.verifyCertificateboolean

      Validate the TLS certificate of the remote host. Specifically the issuer is checked but not CRLs (Certificate Revocation Lists).

      Default: true

    • spec.elasticsearch.tls.verifyHostnameboolean

      Verifies that the name of the remote host matches the name specified in the remote host’s TLS certificate.

      Default: true

• spec.extraLabelsobject

  A set of labels that will be attached to each batch of events.

  You can use simple templating here: {{ app }}.

  There are some reserved keys:

  • parsed_data
  • pod
  • pod_labels_*
  • pod_ip
  • namespace
  • image
  • container
  • node
  • pod_owner

  More about field path notation…

  Example:

  extraLabels:
    forwarder: vector
    key: value
    app_info: "{{ app }}"
    array_member: "{{ array[0] }}"
    symbol_escating_value: "{{ pay\.day }}"
  

• spec.kafkaobject
  • spec.kafka.bootstrapServersarray of strings

    Required value

    A list of host and port pairs that are the addresses of the Kafka brokers in a “bootstrap” Kafka cluster that a Kafka client connects to initially to bootstrap itself.

    Default: []

    • Element of the arraystring

      Pattern: ^(.+)\:\d{1,5}$

  • spec.kafka.tlsobject

    Configures the TLS options for outgoing connections.

    • spec.kafka.tls.caFilestring

      Base64-encoded CA certificate in PEM format.

    • spec.kafka.tls.clientCrtobject

      Configures the client certificate for outgoing connections.

      • spec.kafka.tls.clientCrt.crtFilestring

        Required value

        Base64-encoded certificate in PEM format.

        You must also set the keyFile parameter.

      • spec.kafka.tls.clientCrt.keyFilestring

        Required value

        Base64-encoded private key in PEM format (PKCS#8).

        You must also set the crtFile parameter.

      • spec.kafka.tls.clientCrt.keyPassstring

        Base64-encoded pass phrase used to unlock the encrypted key file.

    • spec.kafka.tls.verifyCertificateboolean

      Validate the TLS certificate of the remote host.

      Default: true

    • spec.kafka.tls.verifyHostnameboolean

      Verifies that the name of the remote host matches the name specified in the remote host’s TLS certificate.

      Default: true

  • spec.kafka.topicstring

    Required value

    The Kafka topic name to write events to. This parameter supports template syntax, which enables you to use dynamic per-event values.

• spec.logstashobject
  • spec.logstash.endpointstring

    Required value

    Base URL of the Logstash instance.

  • spec.logstash.tlsobject

    Configures the TLS options for outgoing connections.

    • spec.logstash.tls.caFilestring

      Base64-encoded CA certificate in PEM format.

    • spec.logstash.tls.clientCrtobject

      Configures the client certificate for outgoing connections.

      • spec.logstash.tls.clientCrt.crtFilestring

        Required value

        Base64-encoded certificate in PEM format.

        You must also set the keyFile parameter.

      • spec.logstash.tls.clientCrt.keyFilestring

        Required value

        Base64-encoded private key in PEM format (PKCS#8).

        You must also set the crtFile parameter.

      • spec.logstash.tls.clientCrt.keyPassstring

        Base64-encoded pass phrase used to unlock the encrypted key file.

    • spec.logstash.tls.verifyCertificateboolean

      Validate the TLS certificate of the remote host.

      Default: true

    • spec.logstash.tls.verifyHostnameboolean

      Verifies that the name of the remote host matches the name specified in the remote host’s TLS certificate.

      Default: true

• spec.lokiobject
  • spec.loki.authobject
    • spec.loki.auth.passwordstring

      Base64-encoded Basic authentication password.

    • spec.loki.auth.strategystring

      The authentication strategy to use.

      Default: "Basic"

      Allowed values: Basic, Bearer

    • spec.loki.auth.tokenstring

      The token to use for Bearer authentication.

    • spec.loki.auth.userstring

      The Basic authentication user name.

  • spec.loki.endpointstring

    Required value

    Base URL of the Loki instance.

    Agent automatically adds /loki/api/v1/push into URL during data transmission.

  • spec.loki.tlsobject

    Configures the TLS options for outgoing connections.

    • spec.loki.tls.caFilestring

      Base64-encoded CA certificate in PEM format.

    • spec.loki.tls.clientCrtobject

      Configures the client certificate for outgoing connections.

      • spec.loki.tls.clientCrt.crtFilestring

        Required value

        Base64-encoded certificate in PEM format.

        You must also set the keyFile parameter.

      • spec.loki.tls.clientCrt.keyFilestring

        Required value

        Base64-encoded private key in PEM format (PKCS#8).

        You must also set the crtFile parameter.

      • spec.loki.tls.clientCrt.keyPassstring

        Base64-encoded pass phrase used to unlock the encrypted key file.

    • spec.loki.tls.verifyHostnameboolean

      Verifies that the name of the remote host matches the name specified in the remote host’s TLS certificate.

      Default: true

• spec.rateLimitobject

  Parameter for limiting the flow of events.

  • spec.rateLimit.linesPerMinutenumber

    Required value

    The number of records per minute.

• spec.splunkobject
  • spec.splunk.endpointstring

    Required value

    Base URL of the Splunk instance.

    Example:

    endpoint: https://http-inputs-hec.splunkcloud.com
    

  • spec.splunk.indexstring

    Index name to write events to.

  • spec.splunk.tlsobject

    Configures the TLS options for outgoing connections.

    • spec.splunk.tls.caFilestring

      Base64-encoded CA certificate in PEM format.

    • spec.splunk.tls.clientCrtobject

      Configures the client certificate for outgoing connections.

      • spec.splunk.tls.clientCrt.crtFilestring

        Required value

        Base64-encoded certificate in PEM format.

        You must also set the keyFile parameter.

      • spec.splunk.tls.clientCrt.keyFilestring

        Required value

        Base64-encoded private key in PEM format (PKCS#8).

        You must also set the crtFile parameter.

      • spec.splunk.tls.clientCrt.keyPassstring

        Base64-encoded pass phrase used to unlock the encrypted key file.

    • spec.splunk.tls.verifyCertificateboolean

      Validate the TLS certificate of the remote host.

      Default: true

    • spec.splunk.tls.verifyHostnameboolean

      Verifies that the name of the remote host matches the name specified in the remote host’s TLS certificate.

      Default: true

  • spec.splunk.tokenstring

    Required value

    Default Splunk HEC token. If an event has a token set in its metadata, it will have priority over the one set here.

• spec.typestring

  Type of a log storage backend.

  Allowed values: Loki, Elasticsearch, Logstash, Vector, Kafka, Splunk

• spec.vectorobject
  • spec.vector.endpointstring

    Required value

    An address of the Vector instance. API v2 must be used for communication between instances.

    Pattern: ^(.+):([0-9]{1,5})$

  • spec.vector.tlsobject

    Configures the TLS options for outgoing connections.

    • spec.vector.tls.caFilestring

      Base64-encoded CA certificate in PEM format.

    • spec.vector.tls.clientCrtobject

      Configures the client certificate for outgoing connections.

      • spec.vector.tls.clientCrt.crtFilestring

        Required value

        Base64-encoded certificate in PEM format.

        You must also set the keyFile parameter.

      • spec.vector.tls.clientCrt.keyFilestring

        Required value

        Base64-encoded private key in PEM format (PKCS#8).

        You must also set the crtFile parameter.

      • spec.vector.tls.clientCrt.keyPassstring

        Base64-encoded passphrase used to unlock the encrypted key file.

    • spec.vector.tls.verifyCertificateboolean

      Validate the TLS certificate of the remote host.

      Default: true

    • spec.vector.tls.verifyHostnameboolean

      Verifies that the name of the remote host matches the name specified in the remote host’s TLS certificate.

      Default: true

Required value

• spec.destinationRefsarray of strings

  Required value

  Array of ClusterLogDestination CustomResource names which this source will output with.

  Fields with float or boolean values will be converted to strings.

• spec.fileobject
  • spec.file.excludearray of strings

    Array of file patterns to exclude.

    Example:

    exclude:
    - "/var/log/nginx/error.log"
    - "/var/log/audit.log"
    

  • spec.file.includearray of strings

    Array of file patterns to include.

    Example:

    include:
    - "/var/log/*.log"
    - "/var/log/nginx/*.log"
    

  • spec.file.lineDelimiterstring

    String sequence used to separate one file line from another.

    Example:

    lineDelimiter: "\\r\\n"
    

• spec.kubernetesPodsobject
  • spec.kubernetesPods.labelSelectorobject

    Specifies the label selector to filter Pods with.

    You can get more into here.

    • spec.kubernetesPods.labelSelector.matchExpressionsarray of objects

      List of label expressions for Pods.

      Example:

      matchExpressions:
      - key: tier
        operator: In
        values:
        - production
        - staging
      

      • spec.kubernetesPods.labelSelector.matchExpressions.keystring

        Required value

      • spec.kubernetesPods.labelSelector.matchExpressions.operatorstring

        Required value

        Allowed values: In, NotIn, Exists, DoesNotExist

      • spec.kubernetesPods.labelSelector.matchExpressions.valuesarray of strings
    • spec.kubernetesPods.labelSelector.matchLabelsobject

      List of labels which Pod should have.

      Example:

      matchLabels:
        foo: bar
        baz: who
      

  • spec.kubernetesPods.namespaceSelectorobject

    Specifies the Namespace selector to filter Pods with.

    • spec.kubernetesPods.namespaceSelector.excludeNamesarray of strings

      Include all namespaces except a particular set.

    • spec.kubernetesPods.namespaceSelector.labelSelectorobject

      Specifies the label selector to filter namespaces.

      You can get more into here.

      • spec.kubernetesPods.namespaceSelector.labelSelector.matchExpressionsarray of objects

        List of label expressions for namespaces.

        Example:

        matchExpressions:
        - key: tier
          operator: In
          values:
          - production
          - staging
        

        • spec.kubernetesPods.namespaceSelector.labelSelector.matchExpressions.keystring

          Required value

        • spec.kubernetesPods.namespaceSelector.labelSelector.matchExpressions.operatorstring

          Required value

          Allowed values: In, NotIn, Exists, DoesNotExist

        • spec.kubernetesPods.namespaceSelector.labelSelector.matchExpressions.valuesarray of strings
      • spec.kubernetesPods.namespaceSelector.labelSelector.matchLabelsobject

        List of labels which a namespace should have.

        Example:

        matchLabels:
          foo: bar
          baz: who
        

    • spec.kubernetesPods.namespaceSelector.matchNamesarray of strings

      Include only a particular set of namespaces.

• spec.labelFilterarray of objects

  Rules to filter log lines by their labels.

  Example:

  labelFilter:
  - field: container
    operator: In
    values:
    - nginx
  - field: pod_labels.tier
    operator: Regex
    values:
    - prod-.+
    - stage-.+
  

  • spec.labelFilter.fieldstring

    Required value

    Label name for filtering. Must not be empty.

    Pattern: .+

  • spec.labelFilter.operatorstring

    Required value

    Operator for log field comparations:

    • In — finds a substring in a string.
    • NotIn — is a negative version of the In operator.
    • Regex — is trying to match regexp over the field; only log events with matching fields will pass.
    • NotRegex — is a negative version of the Regex operator; log events without fields or with not matched fields will pass.
    • Exists — drops log event if it contains some fields.
    • DoesNotExist — drops log event if it does not contain some fields.

    Allowed values: In, NotIn, Regex, NotRegex, Exists, DoesNotExist

  • spec.labelFilter.valuesarray

    Array of values or regexes for corresponding operations. Does not work for Exists and DoesNotExist operations.

    Fields a with float or boolean values will be converted to strings during comparison.

• spec.logFilterarray of objects

  List of filter for logs.

  Only matched lines would be stored to log destination.

  Example:

  logFilter:
  - field: tier
    operator: Exists
  - field: foo
    operator: NotIn
    values:
    - dev
    - 42
    - "true"
    - "3.14"
  - field: bar
    operator: Regex
    values:
    - ^abc
    - ^\d.+$
  

  • spec.logFilter.fieldstring

    Required value

    Field name for filtering. It should be empty for non-JSON messages.

  • spec.logFilter.operatorstring

    Required value

    Operator for log field comparations:

    • In — finds a substring in a string.
    • NotIn — is a negative version of the In operator.
    • Regex — is trying to match regexp over the field; only log events with matching fields will pass.
    • NotRegex — is a negative version of the Regex operator; log events without fields or with not matched fields will pass.
    • Exists — drops log event if it contains some fields.
    • DoesNotExist — drops log event if it does not contain some fields.

    Allowed values: In, NotIn, Regex, NotRegex, Exists, DoesNotExist

  • spec.logFilter.valuesarray

    Array of values or regexes for corresponding operations. Does not work for Exists and DoesNotExist operations.

    Fields a with float or boolean values will be converted to strings during comparison.

• spec.multilineParserobject

  Multiline parser for different patterns

  • spec.multilineParser.typestring

    Required value

    Parser types:

    • None — do not parse logs.
    • General — tries to match general multiline logs with space or tabulation on extra lines.
    • Backslash — tries to match bash style logs with backslash on all lines except the last event line.
    • LogWithTime — tries to detect events by timestamp.
    • MultilineJSON — tries to match JSON logs, assuming the event starts with the { symbol.

    Default: "None"

    Allowed values: None, General, Backslash, LogWithTime, MultilineJSON

• spec.typestring

  Required value

  Set on of possible input sources.

  KubernetesPods source reads logs from Kubernetes Pods.

  File source reads local file from node filesystem.

  Allowed values: KubernetesPods, File

Required value

• spec.clusterDestinationRefsarray of strings

  Required value

  Array of ClusterLogDestination CustomResource names which this source will output with.

• spec.labelFilterarray of objects

  Rules to filter log lines by their labels.

  Example:

  labelFilter:
  - field: container
    operator: In
    values:
    - nginx
  - field: pod_labels.tier
    operator: Regex
    values:
    - prod-.+
    - stage-.+
  

  • spec.labelFilter.fieldstring

    Required value

    Label name for filtering. Must not be empty.

    Pattern: .+

  • spec.labelFilter.operatorstring

    Required value

    Operator for log field comparations:

    • In — finds a substring in a string.
    • NotIn — is a negative version of the In operator.
    • Regex — is trying to match regexp over the field; only log events with matching fields will pass.
    • NotRegex — is a negative version of the Regex operator; log events without fields or with not matched fields will pass.
    • Exists — drops log event if it contains some fields.
    • DoesNotExist — drops log event if it does not contain some fields.

    Allowed values: In, NotIn, Regex, NotRegex, Exists, DoesNotExist

  • spec.labelFilter.valuesarray

    Array of values or regexes for corresponding operations. Does not work for Exists and DoesNotExist operations.

    Fields with a float or boolean values will be converted to strings during comparison.

• spec.labelSelectorobject

  Specifies the label selector to filter Pods.

  You can get more into here.

  • spec.labelSelector.matchExpressionsarray of objects

    List of label expressions for Pods.

    Example:

    matchExpressions:
    - key: tier
      operator: In
      values:
      - production
      - staging
    

    • spec.labelSelector.matchExpressions.keystring

      Required value

    • spec.labelSelector.matchExpressions.operatorstring

      Required value

      Allowed values: In, NotIn, Exists, DoesNotExist

    • spec.labelSelector.matchExpressions.valuesarray of strings
  • spec.labelSelector.matchLabelsobject

    List of labels which Pod should have.

    Example:

    matchLabels:
      foo: bar
      baz: who
    

• spec.logFilterarray of objects

  List of filter for logs.

  Only matched lines would be stored to log destination.

  Example:

  logFilter:
  - field: tier
    operator: Exists
  - field: foo
    operator: NotIn
    values:
    - dev
    - 42
    - "true"
    - "3.14"
  - field: bar
    operator: Regex
    values:
    - ^abc
    - ^\d.+$
  

  • spec.logFilter.fieldstring

    Required value

    Field name for filtering. It should be empty for non-JSON messages.

  • spec.logFilter.operatorstring

    Required value

    Operator for log field comparations:

    • In — finds a substring in a string.
    • NotIn — is a negative version of the In operator.
    • Regex — is trying to match regexp over the field; only log events with matching fields will pass.
    • NotRegex — is a negative version of the Regex operator; log events without fields or with not matched fields will pass.
    • Exists — drops log event if it contains some fields.
    • DoesNotExist — drops log event if it does not contain some fields.

    Allowed values: In, NotIn, Regex, NotRegex, Exists, DoesNotExist

  • spec.logFilter.valuesarray

    Array of values or regexes for corresponding operations. Does not work for Exists and DoesNotExist operations.

    Fields a with float or boolean values will be converted to strings during comparison.

• spec.multilineParserobject

  Multiline parser for different patterns.

  • spec.multilineParser.typestring

    Required value

    Parser types:

    • None — do not parse logs.
    • General — tries to match general multiline logs with space or tabulation on extra lines.
    • Backslash — tries to match bash style logs with backslash on all lines except the last event line.
    • LogWithTime — tries to detect events by timestamp.
    • MultilineJSON — tries to match JSON logs, assuming the event starts with the { symbol.

    Default: "None"

    Allowed values: None, General, Backslash, LogWithTime, MultilineJSON