The log-shipper module: Custom Resources

ClusterLogDestination

Scope: Cluster
Version: v1alpha1

CustomResource for storage in log-pipeline.

Each CR ClusterLogDestination descride one log storage, which you can use in many log sources.

metadata.name — is an upstream name, whick you should use in CR ClusterLoggingConfig.

specobject
Required value
- elasticsearchobject
  - authobject
    - awsAccessKeystring
      Base64 encoded AWS ACCESS_KEY.
    - awsAssumeRolestring
      The ARN of an IAM role to assume at startup.
    - awsRegionstring
      AWS region for aws authentication.
    - awsSecretKeystring
      Base64 encoded AWS SECRET_KEY.
    - passwordstring
      Base64 encoded basic authentication password.
    - strategystring
      The authentication strategy to use.
      
      Default: "Basic"
      Allowed values: Basic, AWS
    - userstring
      The basic authentication user name.
  - dataStreamEnabledboolean
    Use for storage indexes or datastreams (https://www.elastic.co/guide/en/elasticsearch/reference/master/data-streams.html).
    
    Datastream usage is better for logs and metrics storage but they works only for Elasticsearch >= 7.16.X.
    
    Default: false
  - docTypestring
    The doc_type for your index data. This is only relevant for Elasticsearch <= 6.X.
    
    For ES >= 7.X — you do not need this option since this version has removed doc_type mapping.
    
    For ES >= 6.X — preferred value is a ‘_doc’, this value will be easier to upgrade to 7.X.
    
    For ES < 6.X — you can’t use a value starting with ‘_’ or empty string. So use some other value like ‘vector’ or ‘logs’.
  - endpointstring
    Required value
    The base URL of the Elasticsearch instance.
  - indexstring
    Index name to write events to.
  - pipelinestring
    Name of the pipeline to apply.
  - tlsobject
    Configures the TLS options for outgoing connections.
    - caFilestring
      Base64 encoded CA certificate in PEM format.
    - clientCrtobject
      Configures client certificate for outgoing connections.
      - crtFilestring
        Required value
        Base64 encoded certificate in PEM format. keyFile Must also be set.
      - keyFilestring
        Required value
        Base64 encoded private key in PEM format (PKCS#8). If this is set, crtFile must also be set.
      - keyPassstring
        Base64 encoded pass phrase used to unlock the encrypted key file.
    - verifyHostnameboolean
      Vector will validate the TLS certificate of the remote host.
      
      Default: true
- extraLabelsobject
  A set of labels that will be attached to each batch of events. You can use simple templating here: {{ app }}. There are some reserved keys:
  - parsed_data
  - pod
  - pod_labels
  - pod_ip
  - namespace
  - image
  - container
  - node
  - pod_owner More about field path notation
  Example:
```
extraLabels:
  forwarder: vector
  key: value
  app_info: "{{ app }}"
  array_member: "{{ array[0] }}"
  symbol_escating_value: "{{ pay\.day }}"
```
- logstashobject
  - endpointstring
    Required value
    The base URL of the Loki instance.
  - tlsobject
    Configures the TLS options for outgoing connections.
    - caFilestring
      Base64 encoded CA certificate in PEM format.
    - clientCrtobject
      Configures client certificate for outgoing connections.
      - crtFilestring
        Required value
        Base64 encoded certificate in PEM format. keyFile Must also be set.
      - keyFilestring
        Required value
        Base64 encoded private key in PEM format (PKCS#8). If this is set, crtFile must also be set.
      - keyPassstring
        Base64 encoded pass phrase used to unlock the encrypted key file.
    - verifyCertificateboolean
      Vector will validate the configured remote host name against the remote host’s TLS certificate.
      
      Default: true
    - verifyHostnameboolean
      Vector will validate the TLS certificate of the remote host.
      
      Default: true
- lokiobject
  - authobject
    - passwordstring
      Base64 encoded basic authentication password.
    - strategystring
      The authentication strategy to use.
      
      Default: "Basic"
      Allowed values: Basic, Bearer
    - tokenstring
      The token to use for bearer authentication.
    - userstring
      The basic authentication user name.
  - endpointstring
    Required value
    The base URL of the Loki instance.
    
    HINT Vector automatically adds /loki/api/v1/push into URL during data transmission.
  - tlsobject
    Configures the TLS options for outgoing connections.
    - caFilestring
      Base64 encoded CA certificate in PEM format.
    - clientCrtobject
      Configures client certificate for outgoing connections.
      - crtFilestring
        Required value
        Base64 encoded certificate in PEM format. keyFile Must also be set.
      - keyFilestring
        Required value
        Base64 encoded private key in PEM format (PKCS#8). If this is set, crtFile must also be set.
      - keyPassstring
        Base64 encoded pass phrase used to unlock the encrypted key file.
    - verifyHostnameboolean
      Vector will validate the TLS certificate of the remote host.
      
      Default: true
- typestring
  Set on of possible output destinations.
  
  Allowed values: Loki, Elasticsearch, Logstash

ClusterLoggingConfig

Scope: Cluster
Version: v1alpha1

CustomResource for source in log-pipeline.

Each CustomResource ClusterLoggingConfig describes rules for log fetching from cluster.

specobject
Required value
- destinationRefsarray of strings
  Required value
  Array of ClusterLogDestination CustomResource names which this source will output with. Fields with float or boolean values will be converted to string.
- fileobject
  - excludearray of strings
    Array of file patterns to exclude.
    
    Example:
    exclude: - "/var/log/nginx/error.log" - "/var/log/audit.log"
  - includearray of strings
    Array of file patterns to include.
    
    Example:
    include: - "/var/log/*.log" - "/var/log/nginx/*.log"
  - lineDelimiterstring
    String sequence used to separate one file line from another.
    
    Example:
    lineDelimiter: "\\r\\n"
- kubernetesPodsobject
  - labelSelectorobject
    Specifies the label selector to filter Pods with.
    
    You can get more into here.
    - matchExpressionsarray of objects
      List of label expressions for Pods.
      
      Example:
      matchExpressions: | ```yaml matchExpressions: - key: tier operator": In values: - production - staging
```
- keystring
  Required value
- operatorstring
  Required value
  Allowed values: In, NotIn, Exists, DoesNotExist
- valuesarray of strings
</li>
- matchLabelsobject
  List of labels which Pod should have.
  
  Example:
  matchLabels: foo: bar baz: whooo
- namespaceSelectorobject
  Specifies the Namespace selector to filter Pods with.
  - matchNamesarray of strings
- logFilterarray of objects
  List of filter for logs. Only matched lines would be stored to log destination.
  
  Example:
  logFilter: "```yaml\nlogFilter:\n- field: tier\n operator: Exists\n- field: tier\n \ operator: In\n values:\n - production\n - staging\n- field: foo\n operator: NotIn\n values: \n - dev\n - 42\n - \"true\"\n - \"3.14\"\n- field: bar\n operator: Regex\n values:\n - ^abc\n - ^\\d.+$\n```\n"
  - fieldstring
    Required value
    Field name for filtering.
  - operatorstring
    Required value
    Operator for log field comparations:
    
    In operator find substring in string
    
    NotIn operator is negative version of In operator
    
    Regex operator is trying to match regexp over field, only log events with matching fields will pass
    
    NotRegex operator is negative version of Regex operator, log events without fields or with not matched fields will pass
    
    Exists operator drop log event if it contains some fields
    
    DoesNotExist operator drop log event if it does not contain some fields
    
    Allowed values: In, NotIn, Regex, NotRegex, Exists, DoesNotExist
  - valuesarray
    Array of ClusterLogDestination CustomResource names which this source will output with. Fields with float or boolean values will be converted to strings during comparation.
- multilineParserobject
  Multiline parser for different patterns
  - typestring
    Required value
    Parser types:
    
    None - Do not parse logs.
    
    General - Try to match general multiline logs with space or tabulation on extra lines.
    
    Backslash - Try to match bash style logs with backslash on all lines except last event line.
    
    LogWithTime - Try to detect events by timestamps.
    
    MultilineJSON - Try to match json logs by { as a first symbol.
    
    Default: "None"
    Allowed values: None, General, Backslash, LogWithTime, MultilineJSON
- typestring
  Required value
  Set on of possible input sources.
  
  KubernetesPods source reads logs from Kubernetes Pods.
  
  File source reads local file from node filesystem.
  
  Allowed values: KubernetesPods, File

The log-shipper module: Custom Resources

ClusterLogDestination

ClusterLoggingConfig

PodLoggingConfig