ClusterLogDestination

Scope: Cluster
Version: v1alpha1

CustomResource for storage in log-pipeline.

Each CR ClusterLogDestination descride one log storage, which you can use in many log sources.

metadata.name — is an upstream name, whick you should use in CR ClusterLoggingConfig.

  • specobject

    Required value

    • elasticsearchobject
      • authobject
        • awsAccessKeystring

          Base64 encoded AWS ACCESS_KEY.

        • awsAssumeRolestring

          The ARN of an IAM role to assume at startup.

        • awsRegionstring

          AWS region for aws authentication.

        • awsSecretKeystring

          Base64 encoded AWS SECRET_KEY.

        • passwordstring

          Base64 encoded basic authentication password.

        • strategystring

          The authentication strategy to use.

          Default: "Basic"

          Allowed values: Basic, AWS

        • userstring

          The basic authentication user name.

      • dataStreamEnabledboolean

        Use for storage indexes or datastreams (https://www.elastic.co/guide/en/elasticsearch/reference/master/data-streams.html).

        Datastream usage is better for logs and metrics storage but they works only for Elasticsearch >= 7.16.X.

        Default: false

      • docTypestring

        The doc_type for your index data. This is only relevant for Elasticsearch <= 6.X.

        For ES >= 7.X — you do not need this option since this version has removed doc_type mapping.

        For ES >= 6.X — preferred value is a ‘_doc’, this value will be easier to upgrade to 7.X.

        For ES < 6.X — you can’t use a value starting with ‘_’ or empty string. So use some other value like ‘vector’ or ‘logs’.

      • endpointstring

        Required value

        The base URL of the Elasticsearch instance.

      • indexstring

        Index name to write events to.

      • pipelinestring

        Name of the pipeline to apply.

      • tlsobject

        Configures the TLS options for outgoing connections.

        • caFilestring

          Base64 encoded CA certificate in PEM format.

        • clientCrtobject

          Configures client certificate for outgoing connections.

          • crtFilestring

            Required value

            Base64 encoded certificate in PEM format. keyFile Must also be set.

          • keyFilestring

            Required value

            Base64 encoded private key in PEM format (PKCS#8). If this is set, crtFile must also be set.

          • keyPassstring

            Base64 encoded pass phrase used to unlock the encrypted key file.

        • verifyHostnameboolean

          Vector will validate the TLS certificate of the remote host.

          Default: true

    • extraLabelsobject

      A set of labels that will be attached to each batch of events. You can use simple templating here: {{ app }}. There are some reserved keys:

      • parsed_data
      • pod
      • pod_labels
      • pod_ip
      • namespace
      • image
      • container
      • node
      • pod_owner More about field path notation

      Example:

      extraLabels:
        forwarder: vector
        key: value
        app_info: "{{ app }}"
        array_member: "{{ array[0] }}"
        symbol_escating_value: "{{ pay\.day }}"
      
    • logstashobject
      • endpointstring

        Required value

        The base URL of the Loki instance.

      • tlsobject

        Configures the TLS options for outgoing connections.

        • caFilestring

          Base64 encoded CA certificate in PEM format.

        • clientCrtobject

          Configures client certificate for outgoing connections.

          • crtFilestring

            Required value

            Base64 encoded certificate in PEM format. keyFile Must also be set.

          • keyFilestring

            Required value

            Base64 encoded private key in PEM format (PKCS#8). If this is set, crtFile must also be set.

          • keyPassstring

            Base64 encoded pass phrase used to unlock the encrypted key file.

        • verifyCertificateboolean

          Vector will validate the configured remote host name against the remote host’s TLS certificate.

          Default: true

        • verifyHostnameboolean

          Vector will validate the TLS certificate of the remote host.

          Default: true

    • lokiobject
      • authobject
        • passwordstring

          Base64 encoded basic authentication password.

        • strategystring

          The authentication strategy to use.

          Default: "Basic"

          Allowed values: Basic, Bearer

        • tokenstring

          The token to use for bearer authentication.

        • userstring

          The basic authentication user name.

      • endpointstring

        Required value

        The base URL of the Loki instance.

        HINT Vector automatically adds /loki/api/v1/push into URL during data transmission.

      • tlsobject

        Configures the TLS options for outgoing connections.

        • caFilestring

          Base64 encoded CA certificate in PEM format.

        • clientCrtobject

          Configures client certificate for outgoing connections.

          • crtFilestring

            Required value

            Base64 encoded certificate in PEM format. keyFile Must also be set.

          • keyFilestring

            Required value

            Base64 encoded private key in PEM format (PKCS#8). If this is set, crtFile must also be set.

          • keyPassstring

            Base64 encoded pass phrase used to unlock the encrypted key file.

        • verifyHostnameboolean

          Vector will validate the TLS certificate of the remote host.

          Default: true

    • typestring

      Set on of possible output destinations.

      Allowed values: Loki, Elasticsearch, Logstash

ClusterLoggingConfig

Scope: Cluster
Version: v1alpha1

CustomResource for source in log-pipeline.

Each CustomResource ClusterLoggingConfig describes rules for log fetching from cluster.

  • specobject

    Required value

    • destinationRefsarray of strings

      Required value

      Array of ClusterLogDestination CustomResource names which this source will output with. Fields with float or boolean values will be converted to string.

    • fileobject
      • excludearray of strings

        Array of file patterns to exclude.

        Example:

        exclude:
        - "/var/log/nginx/error.log"
        - "/var/log/audit.log"
        
      • includearray of strings

        Array of file patterns to include.

        Example:

        include:
        - "/var/log/*.log"
        - "/var/log/nginx/*.log"
        
      • lineDelimiterstring

        String sequence used to separate one file line from another.

        Example:

        lineDelimiter: "\\r\\n"
        
    • kubernetesPodsobject
      • labelSelectorobject

        Specifies the label selector to filter Pods with.

        You can get more into here.

        • matchExpressionsarray of objects

          List of label expressions for Pods.

          Example:

          matchExpressions: |
            ```yaml
            matchExpressions:
            - key: tier
              operator": In
              values:
              - production
              - staging
          

          ```

          • keystring

            Required value

          • operatorstring

            Required value

            Allowed values: In, NotIn, Exists, DoesNotExist

          • valuesarray of strings
          </li>
        • matchLabelsobject

          List of labels which Pod should have.

          Example:

          matchLabels:
            foo: bar
            baz: whooo
          
        • </ul></li>
        • namespaceSelectorobject

          Specifies the Namespace selector to filter Pods with.

          • matchNamesarray of strings
        • </ul></li>
        • logFilterarray of objects

          List of filter for logs. Only matched lines would be stored to log destination.

          Example:

          logFilter: "```yaml\nlogFilter:\n- field: tier\n  operator: Exists\n- field: tier\n
            \ operator: In\n  values:\n  - production\n  - staging\n- field: foo\n  operator:
            NotIn\n  values: \n  - dev\n  - 42\n  - \"true\"\n  - \"3.14\"\n- field: bar\n  operator:
            Regex\n  values:\n  - ^abc\n  - ^\\d.+$\n```\n"
          
          • fieldstring

            Required value

            Field name for filtering.

          • operatorstring

            Required value

            Operator for log field comparations:

            • In operator find substring in string
            • NotIn operator is negative version of In operator
            • Regex operator is trying to match regexp over field, only log events with matching fields will pass
            • NotRegex operator is negative version of Regex operator, log events without fields or with not matched fields will pass
            • Exists operator drop log event if it contains some fields
            • DoesNotExist operator drop log event if it does not contain some fields

            Allowed values: In, NotIn, Regex, NotRegex, Exists, DoesNotExist

          • valuesarray

            Array of ClusterLogDestination CustomResource names which this source will output with. Fields with float or boolean values will be converted to strings during comparation.

        • multilineParserobject

          Multiline parser for different patterns

          • typestring

            Required value

            Parser types:

            • None - Do not parse logs.
            • General - Try to match general multiline logs with space or tabulation on extra lines.
            • Backslash - Try to match bash style logs with backslash on all lines except last event line.
            • LogWithTime - Try to detect events by timestamps.
            • MultilineJSON - Try to match json logs by { as a first symbol.

            Default: "None"

            Allowed values: None, General, Backslash, LogWithTime, MultilineJSON

        • typestring

          Required value

          Set on of possible input sources.

          KubernetesPods source reads logs from Kubernetes Pods.

          File source reads local file from node filesystem.

          Allowed values: KubernetesPods, File

        • </ul></li></ul></div>

          PodLoggingConfig

          Scope: Namespaced
          Version: v1alpha1

          CustomResource for namespaced Kubernetes source in log-pipeline.

          Each CustomResource PodLoggingConfig describes rules for log fetching from specified Namespace.

          • specobject

            Required value

            • clusterDestinationRefsarray of strings

              Required value

              Array of ClusterLogDestination CustomResource names which this source will output with.

            • labelSelectorobject

              Specifies the label selector to filter Pods with.

              You can get more into here.

              • matchExpressionsarray of objects

                List of label expressions for Pods.

                Example:

                matchExpressions: |
                  ```yaml
                  matchExpressions:
                  - key: tier
                    operator": In
                    values:
                    - production
                    - staging
                

                ```

                • keystring

                  Required value

                • operatorstring

                  Required value

                  Allowed values: In, NotIn, Exists, DoesNotExist

                • valuesarray of strings
                </li>
              • matchLabelsobject

                List of labels which Pod should have.

                Example:

                matchLabels:
                  foo: bar
                  baz: whooo
                
              • </ul></li>
              • logFilterarray of objects

                List of filter for logs. Only matched lines would be stored to log destination.

                Example:

                logFilter: "```yaml\nlogFilter:\n- field: tier\n  operator: Exists\n- field: tier\n
                  \ operator: In\n  values:\n  - production\n  - staging\n- field: foo\n  operator:
                  NotIn\n  values: \n  - dev\n  - 42\n  - \"true\"\n  - \"3.14\"\n- field: bar\n  operator:
                  Regex\n  values:\n  - ^abc\n  - ^\\d.+$\n```\n"
                
                • fieldstring

                  Required value

                  Field name for filtering.

                • operatorstring

                  Required value

                  Operator for log field comparations:

                  • In operator find substring in string
                  • NotIn operator is negative version of In operator
                  • Regex operator is trying to match regexp over field, only log events with matching fields will pass
                  • NotRegex operator is negative version of Regex operator, log events without fields or with not matched fields will pass
                  • Exists operator drop log event if it contains some fields
                  • DoesNotExist operator drop log event if it does not contain some fields

                  Allowed values: In, NotIn, Regex, NotRegex, Exists, DoesNotExist

                • valuesarray

                  Array of ClusterLogDestination CustomResource names which this source will output with. Fields with float or boolean values will be converted to strings during comparation.

              • multilineParserobject

                Multiline parser for different patterns

                • typestring

                  Required value

                  Parser types:

                  • None - Do not parse logs.
                  • General - Try to match general multiline logs with space or tabulation on extra lines.
                  • Backslash - Try to match bash style logs with backslash on all lines except last event line.
                  • LogWithTime - Try to detect events by timestamps.
                  • MultilineJSON - Try to match json logs by { as a first symbol.

                  Default: "None"

                  Allowed values: None, General, Backslash, LogWithTime, MultilineJSON

              • </ul></li></ul></div>