Custom resource ClusterLogDestination

ClusterLogDestination

Scope: Cluster
Version: v1alpha1

Describes setting for a log storage, which you can use in many log sources.

metadata.name — is an upstream name, which you should use in custom resource ClusterLoggingConfig.

spec
object
Required value
- spec.buffer
  object
  Buffer parameters.
  - spec.buffer.disk
    object
    Disk buffer parameters.
    - spec.buffer.disk.maxSize
      integer or string
      The maximum size of the buffer on disk. Must be at least ~256MB (268435488 bytes).
      
      You can express size as a plain integer or as a fixed-point number using one of these quantity suffixes: E, P, T, G, M, k, Ei, Pi, Ti, Gi, Mi, Ki.
      
      More about resource quantity:
      
      kubernetes quantity
      
      memory resource units
      
      Pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
      Examples:
      
      maxSize: 512Mi
      
      maxSize: 268435488
  - spec.buffer.memory
    object
    - spec.buffer.memory.maxEvents
      number
      The maximum number of events allowed in the buffer.
  - spec.buffer.type
    string
    Required value
    The type of buffer to use.
    
    Allowed values: Disk, Memory
  - spec.buffer.whenFull
    string
    Event handling behavior when a buffer is full.
    
    Default: "Block"
    Allowed values: DropNewest, Block
- spec.elasticsearch
  object
  - spec.elasticsearch.auth
    object
    - spec.elasticsearch.auth.awsAccessKey
      string
      Base64-encoded AWS ACCESS_KEY.
    - spec.elasticsearch.auth.awsAssumeRole
      string
      The ARN of an IAM role to assume at startup.
    - spec.elasticsearch.auth.awsRegion
      string
      AWS region for authentication.
    - spec.elasticsearch.auth.awsSecretKey
      string
      Base64-encoded AWS SECRET_KEY.
    - spec.elasticsearch.auth.password
      string
      Base64-encoded Basic authentication password.
    - spec.elasticsearch.auth.strategy
      string
      The authentication strategy to use.
      
      Default: "Basic"
      Allowed values: Basic, AWS
    - spec.elasticsearch.auth.user
      string
      The Basic authentication user name.
  - spec.elasticsearch.dataStreamEnabled
    boolean
    Use for storage indexes or datastreams (https://www.elastic.co/guide/en/elasticsearch/reference/master/data-streams.html).
    
    Datastream usage is better for logs and metrics storage but they works only for Elasticsearch >= 7.16.X.
    
    Default: false
  - spec.elasticsearch.docType
    string
    The doc_type for your index data. This is only relevant for Elasticsearch <= 6.X.
    
    For Elasticsearch >= 7.X you do not need this option since this version has removed doc_type mapping;
    
    For Elasticsearch >= 6.X the recommended value is _doc, because using it will make it easy to upgrade to 7.X;
    
    For Elasticsearch < 6.X you can’t use a value starting with _ or empty string. Use, for example, values like logs.
  - spec.elasticsearch.endpoint
    string
    Required value
    Base URL of the Elasticsearch instance.
  - spec.elasticsearch.index
    string
    Index name to write events to.
  - spec.elasticsearch.pipeline
    string
    Name of the pipeline to apply.
  - spec.elasticsearch.tls
    object
    Configures the TLS options for outgoing connections.
    - spec.elasticsearch.tls.caFile
      string
      Base64-encoded CA certificate in PEM format.
    - spec.elasticsearch.tls.clientCrt
      object
      Configures the client certificate for outgoing connections.
      - spec.elasticsearch.tls.clientCrt.crtFile
        string
        Required value
        Base64-encoded certificate in PEM format.
        
        You must also set the keyFile parameter.
      - spec.elasticsearch.tls.clientCrt.keyFile
        string
        Required value
        Base64-encoded private key in PEM format (PKCS#8).
        
        You must also set the crtFile parameter.
      - spec.elasticsearch.tls.clientCrt.keyPass
        string
        Base64-encoded pass phrase used to unlock the encrypted key file.
    - spec.elasticsearch.tls.secretRef
      object
      Reference to a Kubernetes Secret containing the CA certificate (ca.pem), client certificate (crt.pem), private key (key.pem) and key pass (keyPass) in Base64-encoded PEM format. If specified, TLS settings are overridden with values from the secret. Secret should be located in d8-log-shipper namespace and have log-shipper.deckhouse.io/watch-secret: true label.
      - spec.elasticsearch.tls.secretRef.name
        string
        Name of the Secret with TLS certificates.
    - spec.elasticsearch.tls.verifyCertificate
      boolean
      Validate the TLS certificate of the remote host. Specifically the issuer is checked but not CRLs (Certificate Revocation Lists).
      
      Default: true
    - spec.elasticsearch.tls.verifyHostname
      boolean
      Verifies that the name of the remote host matches the name specified in the remote host’s TLS certificate.
      
      Default: true
- spec.extraLabels
  object
  A set of labels that will be attached to each batch of events.
  
  You can use simple templating here: {{ app }}.
  
  There are some reserved keys:
  - parsed_data
  - pod
  - pod_labels_*
  - pod_ip
  - namespace
  - image
  - container
  - node
  - pod_owner
  More about field path notation…
  Example:
  extraLabels: forwarder: vector key: value app_info: "{{ app }}" array_member: "{{ array[0] }}" symbol_escating_value: "{{ pay\\.day }}"
- spec.kafka
  object
  - spec.kafka.bootstrapServers
    array of strings
    Required value
    A list of host and port pairs that are the addresses of the Kafka brokers in a “bootstrap” Kafka cluster that a Kafka client connects to initially to bootstrap itself.
    
    Default: []
    Example:
    bootstrapServers: - 10.14.22.123:9092 - 10.14.23.332:9092
    - Element of the array
      string
      Pattern: ^(.+)\:\d{1,5}$
  - spec.kafka.encoding
    object
    How to encode the message.
    - spec.kafka.encoding.codec
      string
      Default: "JSON"
      Allowed values: JSON, CEF
  - spec.kafka.keyField
    string
    Allows to set the key_field.
    
    Examples:
    keyField: host
    keyField: node
    keyField: namespace
    keyField: parsed_data.app_info
  - spec.kafka.sasl
    object
    Configuration for SASL authentication when interacting with Kafka.
    - spec.kafka.sasl.mechanism
      string
      Required value
      The SASL mechanism to use. Only PLAIN and SCRAM-based mechanisms are supported.
      
      Allowed values: PLAIN, SCRAM-SHA-256, SCRAM-SHA-512
    - spec.kafka.sasl.password
      string
      Required value
      The SASL password.
      
      Example:
      password: qwerty
    - spec.kafka.sasl.username
      string
      Required value
      The SASL username.
      
      Example:
      username: username
  - spec.kafka.tls
    object
    Configures the TLS options for outgoing connections.
    - spec.kafka.tls.caFile
      string
      Base64-encoded CA certificate in PEM format.
    - spec.kafka.tls.clientCrt
      object
      Configures the client certificate for outgoing connections.
      - spec.kafka.tls.clientCrt.crtFile
        string
        Required value
        Base64-encoded certificate in PEM format.
        
        You must also set the keyFile parameter.
      - spec.kafka.tls.clientCrt.keyFile
        string
        Required value
        Base64-encoded private key in PEM format (PKCS#8).
        
        You must also set the crtFile parameter.
      - spec.kafka.tls.clientCrt.keyPass
        string
        Base64-encoded pass phrase used to unlock the encrypted key file.
    - spec.kafka.tls.secretRef
      object
      Reference to a Kubernetes Secret containing the CA certificate (ca.pem), client certificate (crt.pem), private key (key.pem) and key pass (keyPass) in Base64-encoded PEM format. If specified, TLS settings are overridden with values from the secret. Secret should be located in d8-log-shipper namespace and have log-shipper.deckhouse.io/watch-secret: true label.
      - spec.kafka.tls.secretRef.name
        string
        Name of the Secret with TLS certificates.
    - spec.kafka.tls.verifyCertificate
      boolean
      Validate the TLS certificate of the remote host.
      
      Default: true
    - spec.kafka.tls.verifyHostname
      boolean
      Verifies that the name of the remote host matches the name specified in the remote host’s TLS certificate.
      
      Default: true
  - spec.kafka.topic
    string
    Required value
    The Kafka topic name to write events to. This parameter supports template syntax, which enables you to use dynamic per-event values.
    
    Examples:
    topic: logs
    topic: logs-{{unit}}-%Y-%m-%d
- spec.logstash
  object
  - spec.logstash.endpoint
    string
    Required value
    Base URL of the Logstash instance.
  - spec.logstash.tls
    object
    Configures the TLS options for outgoing connections.
    - spec.logstash.tls.caFile
      string
      Base64-encoded CA certificate in PEM format.
    - spec.logstash.tls.clientCrt
      object
      Configures the client certificate for outgoing connections.
      - spec.logstash.tls.clientCrt.crtFile
        string
        Required value
        Base64-encoded certificate in PEM format.
        
        You must also set the keyFile parameter.
      - spec.logstash.tls.clientCrt.keyFile
        string
        Required value
        Base64-encoded private key in PEM format (PKCS#8).
        
        You must also set the crtFile parameter.
      - spec.logstash.tls.clientCrt.keyPass
        string
        Base64-encoded pass phrase used to unlock the encrypted key file.
    - spec.logstash.tls.secretRef
      object
      Reference to a Kubernetes Secret containing the CA certificate (ca.pem), client certificate (crt.pem), private key (key.pem) and key pass (keyPass) in Base64-encoded PEM format. If specified, TLS settings are overridden with values from the secret. Secret should be located in d8-log-shipper namespace and have log-shipper.deckhouse.io/watch-secret: true label.
      - spec.logstash.tls.secretRef.name
        string
        Name of the Secret with TLS certificates.
    - spec.logstash.tls.verifyCertificate
      boolean
      Validate the TLS certificate of the remote host.
      
      Default: true
    - spec.logstash.tls.verifyHostname
      boolean
      Verifies that the name of the remote host matches the name specified in the remote host’s TLS certificate.
      
      Default: true
- spec.loki
  object
  - spec.loki.auth
    object
    - spec.loki.auth.password
      string
      Base64-encoded Basic authentication password.
    - spec.loki.auth.strategy
      string
      The authentication strategy to use.
      
      Default: "Basic"
      Allowed values: Basic, Bearer
    - spec.loki.auth.token
      string
      The token to use for Bearer authentication.
    - spec.loki.auth.user
      string
      The Basic authentication user name.
  - spec.loki.endpoint
    string
    Required value
    Base URL of the Loki instance.
    
    Agent automatically adds /loki/api/v1/push into URL during data transmission.
  - spec.loki.tenantID
    string
    ID of a tenant.
    
    This option is used only for GrafanaCloud. When running Loki locally, a tenant ID is not required.
  - spec.loki.tls
    object
    Configures the TLS options for outgoing connections.
    - spec.loki.tls.caFile
      string
      Base64-encoded CA certificate in PEM format.
    - spec.loki.tls.clientCrt
      object
      Configures the client certificate for outgoing connections.
      - spec.loki.tls.clientCrt.crtFile
        string
        Required value
        Base64-encoded certificate in PEM format.
        
        You must also set the keyFile parameter.
      - spec.loki.tls.clientCrt.keyFile
        string
        Required value
        Base64-encoded private key in PEM format (PKCS#8).
        
        You must also set the crtFile parameter.
      - spec.loki.tls.clientCrt.keyPass
        string
        Base64-encoded pass phrase used to unlock the encrypted key file.
    - spec.loki.tls.secretRef
      object
      Reference to a Kubernetes Secret containing the CA certificate (ca.pem), client certificate (crt.pem), private key (key.pem) and key pass (keyPass) in Base64-encoded PEM format. If specified, TLS settings are overridden with values from the secret. Secret should be located in d8-log-shipper namespace and have log-shipper.deckhouse.io/watch-secret: true label.
      - spec.loki.tls.secretRef.name
        string
        Name of the Secret with TLS certificates.
    - spec.loki.tls.verifyCertificate
      boolean
      Validate the TLS certificate of the remote host.
      
      If set to false, the certificate is not checked in the Certificate Revocation Lists.
      
      Default: true
    - spec.loki.tls.verifyHostname
      boolean
      Verifies that the name of the remote host matches the name specified in the remote host’s TLS certificate.
      
      Default: true
- spec.rateLimit
  object
  Parameter for limiting the flow of events.
  - spec.rateLimit.excludes
    array of objects
    List of excludes for keyField.
    
    Only NOT matched log entries would be rate limited.
    
    Examples:
    excludes: field: tier operator: Exists
    excludes: field: foo operator: NotIn values: - dev - 42 - 'true' - '3.14'
    excludes: field: bar operator: Regex values: - "^abc" - "^\\d.+$"
    - spec.rateLimit.excludes.field
      string
      Required value
      Field name for filtering.
    - spec.rateLimit.excludes.operator
      string
      Required value
      Operator for log field comparations:
      
      In — finds a substring in a string.
      
      NotIn — is a negative version of the In operator.
      
      Regex — is trying to match regexp over the field; only log events with matching fields will pass.
      
      NotRegex — is a negative version of the Regex operator; log events without fields or with not matched fields will pass.
      
      Exists — drops log event if it contains some fields.
      
      DoesNotExist — drops log event if it does not contain some fields.
      
      Allowed values: In, NotIn, Regex, NotRegex, Exists, DoesNotExist
    - spec.rateLimit.excludes.values
      array
      Array of values or regexes for corresponding operations. Does not work for Exists and DoesNotExist operations.
      
      Fields a with float or boolean values will be converted to strings during comparison.
  - spec.rateLimit.keyField
    string
    The name of the log field whose value will be hashed to determine if the event should be rate limited.
  - spec.rateLimit.linesPerMinute
    number
    Required value
    The number of records per minute.
- spec.socket
  object
  - spec.socket.address
    string
    Required value
    Address of the socket.
    
    Pattern: ^.*:[1-9][0-9]+$
  - spec.socket.encoding
    object
    How to encode the message.
    - spec.socket.encoding.codec
      string
      Default: "JSON"
      Allowed values: Text, JSON, Syslog, CEF, GELF
  - spec.socket.mode
    string
    Required value
    Allowed values: TCP, UDP
  - spec.socket.tcp
    object
    - spec.socket.tcp.tls
      object
      Configures the TLS options for outgoing connections.
      - spec.socket.tcp.tls.caFile
        string
        Base64-encoded CA certificate in PEM format.
      - spec.socket.tcp.tls.clientCrt
        object
        Configures the client certificate for outgoing connections.
        
        spec.socket.tcp.tls.clientCrt.crtFile
        string
        Required value
        Base64-encoded certificate in PEM format.
        
        You must also set the keyFile parameter.
        
        spec.socket.tcp.tls.clientCrt.keyFile
        string
        Required value
        Base64-encoded private key in PEM format (PKCS#8).
        
        You must also set the crtFile parameter.
        
        spec.socket.tcp.tls.clientCrt.keyPass
        string
        Base64-encoded pass phrase used to unlock the encrypted key file.
      - spec.socket.tcp.tls.secretRef
        object
        Reference to a Kubernetes Secret containing the CA certificate (ca.pem), client certificate (crt.pem), private key (key.pem) and key pass (keyPass) in Base64-encoded PEM format. If specified, TLS settings are overridden with values from the secret. Secret should be located in d8-log-shipper namespace and have log-shipper.deckhouse.io/watch-secret: true label.
        
        spec.socket.tcp.tls.secretRef.name
        string
        Name of the Secret with TLS certificates.
    - spec.socket.tcp.verifyCertificate
      boolean
      Validate the TLS certificate of the remote host.
      
      If set to false, the certificate is not checked in the Certificate Revocation Lists.
      
      Default: true
    - spec.socket.tcp.verifyHostname
      boolean
      Verifies that the name of the remote host matches the name specified in the remote host’s TLS certificate.
      
      Default: true
- spec.splunk
  object
  - spec.splunk.endpoint
    string
    Required value
    Base URL of the Splunk instance.
    
    Example:
    endpoint: https://http-inputs-hec.splunkcloud.com
  - spec.splunk.index
    string
    Index name to write events to.
  - spec.splunk.tls
    object
    Configures the TLS options for outgoing connections.
    - spec.splunk.tls.caFile
      string
      Base64-encoded CA certificate in PEM format.
    - spec.splunk.tls.clientCrt
      object
      Configures the client certificate for outgoing connections.
      - spec.splunk.tls.clientCrt.crtFile
        string
        Required value
        Base64-encoded certificate in PEM format.
        
        You must also set the keyFile parameter.
      - spec.splunk.tls.clientCrt.keyFile
        string
        Required value
        Base64-encoded private key in PEM format (PKCS#8).
        
        You must also set the crtFile parameter.
      - spec.splunk.tls.clientCrt.keyPass
        string
        Base64-encoded pass phrase used to unlock the encrypted key file.
    - spec.splunk.tls.secretRef
      object
      Reference to a Kubernetes Secret containing the CA certificate (ca.pem), client certificate (crt.pem), private key (key.pem) and key pass (keyPass) in Base64-encoded PEM format. If specified, TLS settings are overridden with values from the secret. Secret should be located in d8-log-shipper namespace and have log-shipper.deckhouse.io/watch-secret: true label.
      - spec.splunk.tls.secretRef.name
        string
        Name of the Secret with TLS certificates.
    - spec.splunk.tls.verifyCertificate
      boolean
      Validate the TLS certificate of the remote host.
      
      Default: true
    - spec.splunk.tls.verifyHostname
      boolean
      Verifies that the name of the remote host matches the name specified in the remote host’s TLS certificate.
      
      Default: true
  - spec.splunk.token
    string
    Required value
    Default Splunk HEC token. If an event has a token set in its metadata, it will have priority over the one set here.
- spec.type
  string
  Type of a log storage backend.
  
  Allowed values: Loki, Elasticsearch, Logstash, Vector, Kafka, Splunk, Socket
- spec.vector
  object
  - spec.vector.endpoint
    string
    Required value
    An address of the Vector instance. API v2 must be used for communication between instances.
    
    Pattern: ^(.+):([0-9]{1,5})$
  - spec.vector.tls
    object
    Configures the TLS options for outgoing connections.
    - spec.vector.tls.caFile
      string
      Base64-encoded CA certificate in PEM format.
    - spec.vector.tls.clientCrt
      object
      Configures the client certificate for outgoing connections.
      - spec.vector.tls.clientCrt.crtFile
        string
        Required value
        Base64-encoded certificate in PEM format.
        
        You must also set the keyFile parameter.
      - spec.vector.tls.clientCrt.keyFile
        string
        Required value
        Base64-encoded private key in PEM format (PKCS#8).
        
        You must also set the crtFile parameter.
      - spec.vector.tls.clientCrt.keyPass
        string
        Base64-encoded passphrase used to unlock the encrypted key file.
    - spec.vector.tls.secretRef
      object
      Reference to a Kubernetes Secret containing the CA certificate (ca.pem), client certificate (crt.pem), private key (key.pem) and key pass (keyPass) in Base64-encoded PEM format. If specified, TLS settings are overridden with values from the secret. Secret should be located in d8-log-shipper namespace and have log-shipper.deckhouse.io/watch-secret: true label.
      - spec.vector.tls.secretRef.name
        string
        Name of the Secret with TLS certificates.
    - spec.vector.tls.verifyCertificate
      boolean
      Validate the TLS certificate of the remote host.
      
      Default: true
    - spec.vector.tls.verifyHostname
      boolean
      Verifies that the name of the remote host matches the name specified in the remote host’s TLS certificate.
      
      Default: true

Custom resource ClusterLogDestination

ClusterLogDestination

Request trial access

Thank you

Error

Request callback

Thank you

Something went wrong

Book your sessions

Thank you

Error

Request demo

Thank you

Error

Get the PCI SSC Compliance Report

Thank you

Error