ClusterLogDestination

Scope: Cluster
Version: v1alpha1

Describes setting for a log storage, which you can use in many log sources.

metadata.name — is an upstream name, which you should use in CustomResource ClusterLoggingConfig.

  • specobject

    Required value

    • spec.elasticsearchobject
      • spec.elasticsearch.authobject
        • spec.elasticsearch.auth.awsAccessKeystring

          Base64-encoded AWS ACCESS_KEY.

        • spec.elasticsearch.auth.awsAssumeRolestring

          The ARN of an IAM role to assume at startup.

        • spec.elasticsearch.auth.awsRegionstring

          AWS region for authentication.

        • spec.elasticsearch.auth.awsSecretKeystring

          Base64-encoded AWS SECRET_KEY.

        • spec.elasticsearch.auth.passwordstring

          Base64-encoded Basic authentication password.

        • spec.elasticsearch.auth.strategystring

          The authentication strategy to use.

          Default: "Basic"

          Allowed values: Basic, AWS

        • spec.elasticsearch.auth.userstring

          The Basic authentication user name.

      • spec.elasticsearch.dataStreamEnabledboolean

        Use for storage indexes or datastreams (https://www.elastic.co/guide/en/elasticsearch/reference/master/data-streams.html).

        Datastream usage is better for logs and metrics storage but they works only for Elasticsearch >= 7.16.X.

        Default: false

      • spec.elasticsearch.docTypestring

        The doc_type for your index data. This is only relevant for Elasticsearch <= 6.X.

        • For Elasticsearch >= 7.X you do not need this option since this version has removed doc_type mapping;
        • For Elasticsearch >= 6.X the recommended value is _doc, because using it will make it easy to upgrade to 7.X;
        • For Elasticsearch < 6.X you can’t use a value starting with _ or empty string. Use, for example, values like logs.
      • spec.elasticsearch.endpointstring

        Required value

        Base URL of the Elasticsearch instance.

      • spec.elasticsearch.indexstring

        Index name to write events to.

      • spec.elasticsearch.pipelinestring

        Name of the pipeline to apply.

      • spec.elasticsearch.tlsobject

        Configures the TLS options for outgoing connections.

        • spec.elasticsearch.tls.caFilestring

          Base64-encoded CA certificate in PEM format.

        • spec.elasticsearch.tls.clientCrtobject

          Configures the client certificate for outgoing connections.

          • spec.elasticsearch.tls.clientCrt.crtFilestring

            Required value

            Base64-encoded certificate in PEM format.

            You must also set the keyFile parameter.

          • spec.elasticsearch.tls.clientCrt.keyFilestring

            Required value

            Base64-encoded private key in PEM format (PKCS#8).

            You must also set the crtFile parameter.

          • spec.elasticsearch.tls.clientCrt.keyPassstring

            Base64-encoded pass phrase used to unlock the encrypted key file.

        • spec.elasticsearch.tls.verifyCertificateboolean

          Validate the TLS certificate of the remote host. Specifically the issuer is checked but not CRLs (Certificate Revocation Lists).

          Default: true

        • spec.elasticsearch.tls.verifyHostnameboolean

          Verifies that the name of the remote host matches the name specified in the remote host’s TLS certificate.

          Default: true

    • spec.extraLabelsobject

      A set of labels that will be attached to each batch of events.

      You can use simple templating here: {{ app }}.

      There are some reserved keys:

      • parsed_data
      • pod
      • pod_labels_*
      • pod_ip
      • namespace
      • image
      • container
      • node
      • pod_owner

      More about field path notation…

      Example:

      extraLabels:
        forwarder: vector
        key: value
        app_info: "{{ app }}"
        array_member: "{{ array[0] }}"
        symbol_escating_value: "{{ pay\.day }}"
      
    • spec.kafkaobject
      • spec.kafka.bootstrapServersarray of strings

        Required value

        A list of host and port pairs that are the addresses of the Kafka brokers in a “bootstrap” Kafka cluster that a Kafka client connects to initially to bootstrap itself.

        Default: []

        • Element of the arraystring

          Pattern: ^(.+)\:\d{1,5}$

      • spec.kafka.tlsobject

        Configures the TLS options for outgoing connections.

        • spec.kafka.tls.caFilestring

          Base64-encoded CA certificate in PEM format.

        • spec.kafka.tls.clientCrtobject

          Configures the client certificate for outgoing connections.

          • spec.kafka.tls.clientCrt.crtFilestring

            Required value

            Base64-encoded certificate in PEM format.

            You must also set the keyFile parameter.

          • spec.kafka.tls.clientCrt.keyFilestring

            Required value

            Base64-encoded private key in PEM format (PKCS#8).

            You must also set the crtFile parameter.

          • spec.kafka.tls.clientCrt.keyPassstring

            Base64-encoded pass phrase used to unlock the encrypted key file.

        • spec.kafka.tls.verifyCertificateboolean

          Validate the TLS certificate of the remote host.

          Default: true

        • spec.kafka.tls.verifyHostnameboolean

          Verifies that the name of the remote host matches the name specified in the remote host’s TLS certificate.

          Default: true

      • spec.kafka.topicstring

        Required value

        The Kafka topic name to write events to. This parameter supports template syntax, which enables you to use dynamic per-event values.

    • spec.logstashobject
      • spec.logstash.endpointstring

        Required value

        Base URL of the Logstash instance.

      • spec.logstash.tlsobject

        Configures the TLS options for outgoing connections.

        • spec.logstash.tls.caFilestring

          Base64-encoded CA certificate in PEM format.

        • spec.logstash.tls.clientCrtobject

          Configures the client certificate for outgoing connections.

          • spec.logstash.tls.clientCrt.crtFilestring

            Required value

            Base64-encoded certificate in PEM format.

            You must also set the keyFile parameter.

          • spec.logstash.tls.clientCrt.keyFilestring

            Required value

            Base64-encoded private key in PEM format (PKCS#8).

            You must also set the crtFile parameter.

          • spec.logstash.tls.clientCrt.keyPassstring

            Base64-encoded pass phrase used to unlock the encrypted key file.

        • spec.logstash.tls.verifyCertificateboolean

          Validate the TLS certificate of the remote host.

          Default: true

        • spec.logstash.tls.verifyHostnameboolean

          Verifies that the name of the remote host matches the name specified in the remote host’s TLS certificate.

          Default: true

    • spec.lokiobject
      • spec.loki.authobject
        • spec.loki.auth.passwordstring

          Base64-encoded Basic authentication password.

        • spec.loki.auth.strategystring

          The authentication strategy to use.

          Default: "Basic"

          Allowed values: Basic, Bearer

        • spec.loki.auth.tokenstring

          The token to use for Bearer authentication.

        • spec.loki.auth.userstring

          The Basic authentication user name.

      • spec.loki.endpointstring

        Required value

        Base URL of the Loki instance.

        Agent automatically adds /loki/api/v1/push into URL during data transmission.

      • spec.loki.tlsobject

        Configures the TLS options for outgoing connections.

        • spec.loki.tls.caFilestring

          Base64-encoded CA certificate in PEM format.

        • spec.loki.tls.clientCrtobject

          Configures the client certificate for outgoing connections.

          • spec.loki.tls.clientCrt.crtFilestring

            Required value

            Base64-encoded certificate in PEM format.

            You must also set the keyFile parameter.

          • spec.loki.tls.clientCrt.keyFilestring

            Required value

            Base64-encoded private key in PEM format (PKCS#8).

            You must also set the crtFile parameter.

          • spec.loki.tls.clientCrt.keyPassstring

            Base64-encoded pass phrase used to unlock the encrypted key file.

        • spec.loki.tls.verifyHostnameboolean

          Verifies that the name of the remote host matches the name specified in the remote host’s TLS certificate.

          Default: true

    • spec.rateLimitobject

      Parameter for limiting the flow of events.

      • spec.rateLimit.linesPerMinutenumber

        Required value

        The number of records per minute.

    • spec.splunkobject
      • spec.splunk.endpointstring

        Required value

        Base URL of the Splunk instance.

        Example:

        endpoint: https://http-inputs-hec.splunkcloud.com
        
      • spec.splunk.indexstring

        Index name to write events to.

      • spec.splunk.tlsobject

        Configures the TLS options for outgoing connections.

        • spec.splunk.tls.caFilestring

          Base64-encoded CA certificate in PEM format.

        • spec.splunk.tls.clientCrtobject

          Configures the client certificate for outgoing connections.

          • spec.splunk.tls.clientCrt.crtFilestring

            Required value

            Base64-encoded certificate in PEM format.

            You must also set the keyFile parameter.

          • spec.splunk.tls.clientCrt.keyFilestring

            Required value

            Base64-encoded private key in PEM format (PKCS#8).

            You must also set the crtFile parameter.

          • spec.splunk.tls.clientCrt.keyPassstring

            Base64-encoded pass phrase used to unlock the encrypted key file.

        • spec.splunk.tls.verifyCertificateboolean

          Validate the TLS certificate of the remote host.

          Default: true

        • spec.splunk.tls.verifyHostnameboolean

          Verifies that the name of the remote host matches the name specified in the remote host’s TLS certificate.

          Default: true

      • spec.splunk.tokenstring

        Required value

        Default Splunk HEC token. If an event has a token set in its metadata, it will have priority over the one set here.

    • spec.typestring

      Type of a log storage backend.

      Allowed values: Loki, Elasticsearch, Logstash, Vector, Kafka, Splunk

    • spec.vectorobject
      • spec.vector.endpointstring

        Required value

        An address of the Vector instance. API v2 must be used for communication between instances.

        Pattern: ^(.+):([0-9]{1,5})$

      • spec.vector.tlsobject

        Configures the TLS options for outgoing connections.

        • spec.vector.tls.caFilestring

          Base64-encoded CA certificate in PEM format.

        • spec.vector.tls.clientCrtobject

          Configures the client certificate for outgoing connections.

          • spec.vector.tls.clientCrt.crtFilestring

            Required value

            Base64-encoded certificate in PEM format.

            You must also set the keyFile parameter.

          • spec.vector.tls.clientCrt.keyFilestring

            Required value

            Base64-encoded private key in PEM format (PKCS#8).

            You must also set the crtFile parameter.

          • spec.vector.tls.clientCrt.keyPassstring

            Base64-encoded passphrase used to unlock the encrypted key file.

        • spec.vector.tls.verifyCertificateboolean

          Validate the TLS certificate of the remote host.

          Default: true

        • spec.vector.tls.verifyHostnameboolean

          Verifies that the name of the remote host matches the name specified in the remote host’s TLS certificate.

          Default: true

ClusterLoggingConfig

Scope: Cluster
Version: v1alpha1

Describes a log source in log-pipeline.

Each CustomResource ClusterLoggingConfig describes rules for log fetching from cluster.

  • specobject

    Required value

    • spec.destinationRefsarray of strings

      Required value

      Array of ClusterLogDestination CustomResource names which this source will output with.

      Fields with float or boolean values will be converted to strings.

    • spec.fileobject
      • spec.file.excludearray of strings

        Array of file patterns to exclude.

        Example:

        exclude:
        - "/var/log/nginx/error.log"
        - "/var/log/audit.log"
        
      • spec.file.includearray of strings

        Array of file patterns to include.

        Example:

        include:
        - "/var/log/*.log"
        - "/var/log/nginx/*.log"
        
      • spec.file.lineDelimiterstring

        String sequence used to separate one file line from another.

        Example:

        lineDelimiter: "\\r\\n"
        
    • spec.kubernetesPodsobject
      • spec.kubernetesPods.labelSelectorobject

        Specifies the label selector to filter Pods with.

        You can get more into here.

        • spec.kubernetesPods.labelSelector.matchExpressionsarray of objects

          List of label expressions for Pods.

          Example:

          matchExpressions:
          - key: tier
            operator: In
            values:
            - production
            - staging
          
          • spec.kubernetesPods.labelSelector.matchExpressions.keystring

            Required value

          • spec.kubernetesPods.labelSelector.matchExpressions.operatorstring

            Required value

            Allowed values: In, NotIn, Exists, DoesNotExist

          • spec.kubernetesPods.labelSelector.matchExpressions.valuesarray of strings
        • spec.kubernetesPods.labelSelector.matchLabelsobject

          List of labels which Pod should have.

          Example:

          matchLabels:
            foo: bar
            baz: who
          
      • spec.kubernetesPods.namespaceSelectorobject

        Specifies the Namespace selector to filter Pods with.

        • spec.kubernetesPods.namespaceSelector.excludeNamesarray of strings

          Include all namespaces except a particular set.

        • spec.kubernetesPods.namespaceSelector.labelSelectorobject

          Specifies the label selector to filter namespaces.

          You can get more into here.

          • spec.kubernetesPods.namespaceSelector.labelSelector.matchExpressionsarray of objects

            List of label expressions for namespaces.

            Example:

            matchExpressions:
            - key: tier
              operator: In
              values:
              - production
              - staging
            
            • spec.kubernetesPods.namespaceSelector.labelSelector.matchExpressions.keystring

              Required value

            • spec.kubernetesPods.namespaceSelector.labelSelector.matchExpressions.operatorstring

              Required value

              Allowed values: In, NotIn, Exists, DoesNotExist

            • spec.kubernetesPods.namespaceSelector.labelSelector.matchExpressions.valuesarray of strings
          • spec.kubernetesPods.namespaceSelector.labelSelector.matchLabelsobject

            List of labels which a namespace should have.

            Example:

            matchLabels:
              foo: bar
              baz: who
            
        • spec.kubernetesPods.namespaceSelector.matchNamesarray of strings

          Include only a particular set of namespaces.

    • spec.labelFilterarray of objects

      Rules to filter log lines by their labels.

      Example:

      labelFilter:
      - field: container
        operator: In
        values:
        - nginx
      - field: pod_labels.tier
        operator: Regex
        values:
        - prod-.+
        - stage-.+
      
      • spec.labelFilter.fieldstring

        Required value

        Label name for filtering. Must not be empty.

        Pattern: .+

      • spec.labelFilter.operatorstring

        Required value

        Operator for log field comparations:

        • In — finds a substring in a string.
        • NotIn — is a negative version of the In operator.
        • Regex — is trying to match regexp over the field; only log events with matching fields will pass.
        • NotRegex — is a negative version of the Regex operator; log events without fields or with not matched fields will pass.
        • Exists — drops log event if it contains some fields.
        • DoesNotExist — drops log event if it does not contain some fields.

        Allowed values: In, NotIn, Regex, NotRegex, Exists, DoesNotExist

      • spec.labelFilter.valuesarray

        Array of values or regexes for corresponding operations. Does not work for Exists and DoesNotExist operations.

        Fields a with float or boolean values will be converted to strings during comparison.

    • spec.logFilterarray of objects

      List of filter for logs.

      Only matched lines would be stored to log destination.

      Example:

      logFilter:
      - field: tier
        operator: Exists
      - field: foo
        operator: NotIn
        values:
        - dev
        - 42
        - "true"
        - "3.14"
      - field: bar
        operator: Regex
        values:
        - ^abc
        - ^\d.+$
      
      • spec.logFilter.fieldstring

        Required value

        Field name for filtering. It should be empty for non-JSON messages.

      • spec.logFilter.operatorstring

        Required value

        Operator for log field comparations:

        • In — finds a substring in a string.
        • NotIn — is a negative version of the In operator.
        • Regex — is trying to match regexp over the field; only log events with matching fields will pass.
        • NotRegex — is a negative version of the Regex operator; log events without fields or with not matched fields will pass.
        • Exists — drops log event if it contains some fields.
        • DoesNotExist — drops log event if it does not contain some fields.

        Allowed values: In, NotIn, Regex, NotRegex, Exists, DoesNotExist

      • spec.logFilter.valuesarray

        Array of values or regexes for corresponding operations. Does not work for Exists and DoesNotExist operations.

        Fields a with float or boolean values will be converted to strings during comparison.

    • spec.multilineParserobject

      Multiline parser for different patterns

      • spec.multilineParser.typestring

        Required value

        Parser types:

        • None — do not parse logs.
        • General — tries to match general multiline logs with space or tabulation on extra lines.
        • Backslash — tries to match bash style logs with backslash on all lines except the last event line.
        • LogWithTime — tries to detect events by timestamp.
        • MultilineJSON — tries to match JSON logs, assuming the event starts with the { symbol.

        Default: "None"

        Allowed values: None, General, Backslash, LogWithTime, MultilineJSON

    • spec.typestring

      Required value

      Set on of possible input sources.

      KubernetesPods source reads logs from Kubernetes Pods.

      File source reads local file from node filesystem.

      Allowed values: KubernetesPods, File

PodLoggingConfig

Scope: Namespaced
Version: v1alpha1

CustomResource for namespaced Kubernetes source.

Each CustomResource PodLoggingConfig describes rules for log fetching from specified Namespace.

  • specobject

    Required value

    • spec.clusterDestinationRefsarray of strings

      Required value

      Array of ClusterLogDestination CustomResource names which this source will output with.

    • spec.labelFilterarray of objects

      Rules to filter log lines by their labels.

      Example:

      labelFilter:
      - field: container
        operator: In
        values:
        - nginx
      - field: pod_labels.tier
        operator: Regex
        values:
        - prod-.+
        - stage-.+
      
      • spec.labelFilter.fieldstring

        Required value

        Label name for filtering. Must not be empty.

        Pattern: .+

      • spec.labelFilter.operatorstring

        Required value

        Operator for log field comparations:

        • In — finds a substring in a string.
        • NotIn — is a negative version of the In operator.
        • Regex — is trying to match regexp over the field; only log events with matching fields will pass.
        • NotRegex — is a negative version of the Regex operator; log events without fields or with not matched fields will pass.
        • Exists — drops log event if it contains some fields.
        • DoesNotExist — drops log event if it does not contain some fields.

        Allowed values: In, NotIn, Regex, NotRegex, Exists, DoesNotExist

      • spec.labelFilter.valuesarray

        Array of values or regexes for corresponding operations. Does not work for Exists and DoesNotExist operations.

        Fields with a float or boolean values will be converted to strings during comparison.

    • spec.labelSelectorobject

      Specifies the label selector to filter Pods.

      You can get more into here.

      • spec.labelSelector.matchExpressionsarray of objects

        List of label expressions for Pods.

        Example:

        matchExpressions:
        - key: tier
          operator: In
          values:
          - production
          - staging
        
        • spec.labelSelector.matchExpressions.keystring

          Required value

        • spec.labelSelector.matchExpressions.operatorstring

          Required value

          Allowed values: In, NotIn, Exists, DoesNotExist

        • spec.labelSelector.matchExpressions.valuesarray of strings
      • spec.labelSelector.matchLabelsobject

        List of labels which Pod should have.

        Example:

        matchLabels:
          foo: bar
          baz: who
        
    • spec.logFilterarray of objects

      List of filter for logs.

      Only matched lines would be stored to log destination.

      Example:

      logFilter:
      - field: tier
        operator: Exists
      - field: foo
        operator: NotIn
        values:
        - dev
        - 42
        - "true"
        - "3.14"
      - field: bar
        operator: Regex
        values:
        - ^abc
        - ^\d.+$
      
      • spec.logFilter.fieldstring

        Required value

        Field name for filtering. It should be empty for non-JSON messages.

      • spec.logFilter.operatorstring

        Required value

        Operator for log field comparations:

        • In — finds a substring in a string.
        • NotIn — is a negative version of the In operator.
        • Regex — is trying to match regexp over the field; only log events with matching fields will pass.
        • NotRegex — is a negative version of the Regex operator; log events without fields or with not matched fields will pass.
        • Exists — drops log event if it contains some fields.
        • DoesNotExist — drops log event if it does not contain some fields.

        Allowed values: In, NotIn, Regex, NotRegex, Exists, DoesNotExist

      • spec.logFilter.valuesarray

        Array of values or regexes for corresponding operations. Does not work for Exists and DoesNotExist operations.

        Fields a with float or boolean values will be converted to strings during comparison.

    • spec.multilineParserobject

      Multiline parser for different patterns.

      • spec.multilineParser.typestring

        Required value

        Parser types:

        • None — do not parse logs.
        • General — tries to match general multiline logs with space or tabulation on extra lines.
        • Backslash — tries to match bash style logs with backslash on all lines except the last event line.
        • LogWithTime — tries to detect events by timestamp.
        • MultilineJSON — tries to match JSON logs, assuming the event starts with the { symbol.

        Default: "None"

        Allowed values: None, General, Backslash, LogWithTime, MultilineJSON