The log-shipper module: Custom Resources

ClusterLogDestination

Scope: Cluster
Version: v1alpha1

Describes setting for a log storage, which you can use in many log sources.

metadata.name — is an upstream name, which you should use in CR ClusterLoggingConfig.

specobject
Required value
- elasticsearchobject
  - authobject
    - awsAccessKeystring
      Base64 encoded AWS ACCESS_KEY.
    - awsAssumeRolestring
      The ARN of an IAM role to assume at startup.
    - awsRegionstring
      AWS region for authentication.
    - awsSecretKeystring
      Base64 encoded AWS SECRET_KEY.
    - passwordstring
      Base64 encoded Basic authentication password.
    - strategystring
      The authentication strategy to use.
      
      Default: "Basic"
      Allowed values: Basic, AWS
    - userstring
      The Basic authentication user name.
  - dataStreamEnabledboolean
    Use for storage indexes or datastreams (https://www.elastic.co/guide/en/elasticsearch/reference/master/data-streams.html).
    
    Datastream usage is better for logs and metrics storage but they works only for Elasticsearch >= 7.16.X.
    
    Default: false
  - docTypestring
    The doc_type for your index data. This is only relevant for Elasticsearch <= 6.X.
    - For Elasticsearch >= 7.X you do not need this option since this version has removed doc_type mapping;
    - For Elasticsearch >= 6.X the recommended value is _doc, because using it will make it easy to upgrade to 7.X;
    - For Elasticsearch < 6.X you can’t use a value starting with _ or empty string. Use, for example, values like logs.
  - endpointstring
    Required value
    The base URL of the Elasticsearch instance.
  - indexstring
    Index name to write events to.
  - pipelinestring
    Name of the pipeline to apply.
  - tlsobject
    Configures the TLS options for outgoing connections.
    - caFilestring
      Base64 encoded CA certificate in PEM format.
    - clientCrtobject
      Configures client certificate for outgoing connections.
      - crtFilestring
        Required value
        Base64 encoded certificate in PEM format.
        
        You must also set the keyFile parameter.
      - keyFilestring
        Required value
        Base64 encoded private key in PEM format (PKCS#8).
        
        You must also set the crtFile parameter.
      - keyPassstring
        Base64 encoded pass phrase used to unlock the encrypted key file.
    - verifyHostnameboolean
      Validate the configured remote host name against the remote host’s TLS certificate.
      
      Default: true
- extraLabelsobject
  A set of labels that will be attached to each batch of events.
  
  You can use simple templating here: {{ app }}.
  
  There are some reserved keys:
  - parsed_data
  - pod
  - pod_labels
  - pod_ip
  - namespace
  - image
  - container
  - node
  - pod_owner
  More about field path notation…
  Example:
```
extraLabels:
  forwarder: vector
  key: value
  app_info: "{{ app }}"
  array_member: "{{ array[0] }}"
  symbol_escating_value: "{{ pay\.day }}"
```
- logstashobject
  - endpointstring
    Required value
    The base URL of the Loki instance.
  - tlsobject
    Configures the TLS options for outgoing connections.
    - caFilestring
      Base64 encoded CA certificate in PEM format.
    - clientCrtobject
      Configures client certificate for outgoing connections.
      - crtFilestring
        Required value
        Base64 encoded certificate in PEM format.
        
        You must also set the keyFile parameter.
      - keyFilestring
        Required value
        Base64 encoded private key in PEM format (PKCS#8).
        
        You must also set the crtFile parameter.
      - keyPassstring
        Base64 encoded pass phrase used to unlock the encrypted key file.
    - verifyCertificateboolean
      Validate the TLS certificate of the remote host.
      
      Default: true
    - verifyHostnameboolean
      Validate the configured remote host name against the remote host’s TLS certificate.
      
      Default: true
- lokiobject
  - authobject
    - passwordstring
      Base64 encoded Basic authentication password.
    - strategystring
      The authentication strategy to use.
      
      Default: "Basic"
      Allowed values: Basic, Bearer
    - tokenstring
      The token to use for Bearer authentication.
    - userstring
      The Basic authentication user name.
  - endpointstring
    Required value
    The base URL of the Loki instance.
    
    Agent automatically adds /loki/api/v1/push into URL during data transmission.
  - tlsobject
    Configures the TLS options for outgoing connections.
    - caFilestring
      Base64 encoded CA certificate in PEM format.
    - clientCrtobject
      Configures client certificate for outgoing connections.
      - crtFilestring
        Required value
        Base64 encoded certificate in PEM format.
        
        You must also set the keyFile parameter.
      - keyFilestring
        Required value
        Base64 encoded private key in PEM format (PKCS#8).
        
        You must also set the crtFile parameter.
      - keyPassstring
        Base64 encoded pass phrase used to unlock the encrypted key file.
    - verifyHostnameboolean
      Validate the configured remote host name against the remote host’s TLS certificate.
      
      Default: true
- typestring
  Type of a log storage backend.
  
  Allowed values: Loki, Elasticsearch, Logstash

ClusterLoggingConfig

Scope: Cluster
Version: v1alpha1

Describes a log source in log-pipeline.

Each CustomResource ClusterLoggingConfig describes rules for log fetching from cluster.

specobject
Required value
- destinationRefsarray of strings
  Required value
  Array of ClusterLogDestination CustomResource names which this source will output with. Fields with float or boolean values will be converted to string.
- fileobject
  - excludearray of strings
    Array of file patterns to exclude.
    
    Example:
    exclude: - "/var/log/nginx/error.log" - "/var/log/audit.log"
  - includearray of strings
    Array of file patterns to include.
    
    Example:
    include: - "/var/log/*.log" - "/var/log/nginx/*.log"
  - lineDelimiterstring
    String sequence used to separate one file line from another.
    
    Example:
    lineDelimiter: "\\r\\n"
- kubernetesPodsobject
  - labelSelectorobject
    Specifies the label selector to filter Pods with.
    
    You can get more into here.
    - matchExpressionsarray of objects
      List of label expressions for Pods.
      
      Example:
      matchExpressions: - key: tier operator: In values: - production - staging
      - keystring
        Required value
      - operatorstring
        Required value
        Allowed values: In, NotIn, Exists, DoesNotExist
      - valuesarray of strings
    - matchLabelsobject
      List of labels which Pod should have.
      
      Example:
      matchLabels: foo: bar baz: who
  - namespaceSelectorobject
    Specifies the Namespace selector to filter Pods with.
    - excludeNamesarray of strings
      Include all namespaces except a particular set.
    - matchNamesarray of strings
      Include only a particular set of namespaces.
- logFilterarray of objects
  List of filter for logs.
  
  Only matched lines would be stored to log destination.
  
  Example:
```
logFilter:
- field: tier
  operator: Exists
- field: tier
  operator: In
  values:
  - production
  - staging
- field: foo
  operator: NotIn
  values:
  - dev
  - 42
  - "true"
  - "3.14"
- field: bar
  operator: Regex
  values:
  - ^abc
  - ^\d.+$
```
  - fieldstring
    Required value
    Field name for filtering.
  - operatorstring
    Required value
    Operator for log field comparations:
    - In — finds a substring in string
    - NotIn — is a negative version of the In operator
    - Regex — is trying to match regexp over field, only log events with matching fields will pass
    - NotRegex — is a negative version of the Regex operator, log events without fields or with not matched fields will pass
    - Exists — drops log event if it contains some fields
    - DoesNotExist — drops log event if it does not contain some fields
    Allowed values: In, NotIn, Regex, NotRegex, Exists, DoesNotExist
  - valuesarray
    Array of ClusterLogDestination CustomResource names which this source will output with.
    
    Fields a with float or boolean values will be converted to strings during comparison.
- multilineParserobject
  Multiline parser for different patterns
  - typestring
    Required value
    Parser types:
    - None - do not parse logs
    - General - tries to match general multiline logs with space or tabulation on extra lines
    - Backslash - tries to match bash style logs with backslash on all lines except the last event line
    - LogWithTime - tries to detect events by timestamp
    - MultilineJSON - tries to match JSON logs, assuming the event starts with the { symbol
    Default: "None"
    Allowed values: None, General, Backslash, LogWithTime, MultilineJSON
- typestring
  Required value
  Set on of possible input sources.
  
  KubernetesPods source reads logs from Kubernetes Pods.
  
  File source reads local file from node filesystem.
  
  Allowed values: KubernetesPods, File

PodLoggingConfig

Scope: Namespaced
Version: v1alpha1

CustomResource for namespaced Kubernetes source in log-pipeline.

Each CustomResource PodLoggingConfig describes rules for log fetching from specified Namespace.

specobject
Required value
- clusterDestinationRefsarray of strings
  Required value
  Array of ClusterLogDestination CustomResource names which this source will output with.
- labelSelectorobject
  Specifies the label selector to filter Pods with.
  
  You can get more into here.
  - matchExpressionsarray of objects
    List of label expressions for Pods.
    
    Example:
    matchExpressions: - key: tier operator: In values: - production - staging
    - keystring
      Required value
    - operatorstring
      Required value
      Allowed values: In, NotIn, Exists, DoesNotExist
    - valuesarray of strings
  - matchLabelsobject
    List of labels which Pod should have.
    
    Example:
    matchLabels: foo: bar baz: whooo
- logFilterarray of objects
  List of filter for logs.
  
  Only matched lines would be stored to log destination.
  
  Example:
```
logFilter:
- field: tier
  operator: Exists
- field: tier
  operator: In
  values:
  - production
  - staging
- field: foo
  operator: NotIn
  values: 
  - dev
  - 42
  - "true"
  - "3.14"
- field: bar
  operator: Regex
  values:
  - ^abc
  - ^\d.+$
```
  - fieldstring
    Required value
    Field name for filtering.
  - operatorstring
    Required value
    Operator for log field comparations:
    - In — finds a substring in string
    - NotIn — is a negative version of the In operator
    - Regex — is trying to match regexp over field, only log events with matching fields will pass
    - NotRegex — is a negative version of the Regex operator, log events without fields or with not matched fields will pass
    - Exists — drops log event if it contains some fields
    - DoesNotExist — drops log event if it does not contain some fields
    Allowed values: In, NotIn, Regex, NotRegex, Exists, DoesNotExist
  - valuesarray
    Array of ClusterLogDestination CustomResource names which this source will output with.
    
    Fields a with float or boolean values will be converted to strings during comparison.
- multilineParserobject
  Multiline parser for different patterns.
  - typestring
    Required value
    Parser types:
    - None - do not parse logs
    - General - tries to match general multiline logs with space or tabulation on extra lines
    - Backslash - tries to match bash style logs with backslash on all lines except the last event line
    - LogWithTime - tries to detect events by timestamp
    - MultilineJSON - tries to match JSON logs, assuming the event starts with the { symbol
    Default: "None"
    Allowed values: None, General, Backslash, LogWithTime, MultilineJSON