The module lifecycle stage: Preview
The module has requirements for installation

This section provides basic examples of publishing applications through the alb module.

Publishing an application through a ClusterALBInstance object

This scenario assumes that the ClusterALBInstance object has already been created by an administrator and has reached the Ready state. The name and namespace of the managed Gateway object should be taken from the status of the ClusterALBInstance object.

Next, create a ListenerSet object that will be bound to the desired gateway (using the spec.parentRef.name parameter) and HTTPRoute objects (routes) to route incoming requests to the application. Example:

apiVersion: gateway.networking.k8s.io/v1
kind: ListenerSet
metadata:
  name: app-listeners
  namespace: prod
spec:
  parentRef:
    name: public-gw   # The name of the Gateway object from the ClusterALBInstance status, provided by the administrator.
    namespace: d8-alb
  listeners:
    - name: app-http
      port: 80 # HTTP traffic always uses 80 regardless of ClusterALBInstance settings.
      protocol: HTTP
      hostname: app.example.com
    - name: app-https
      port: 443 # HTTPS traffic always uses 443 regardless of ClusterALBInstance settings.
      protocol: HTTPS
      hostname: app.example.com
      tls:
        mode: Terminate
        certificateRefs:
          - name: app-tls   # Reference to the secret with the TLS certificate.
            namespace: prod
---
# Route for HTTP traffic
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: app-http-route
  namespace: prod
spec:
  parentRefs:
    - name: app-listeners # ListenerSet name.
      namespace: prod
      kind: ListenerSet
      group: gateway.networking.k8s.io
      sectionName: app-http
      port: 80 # HTTP traffic always uses 80 regardless of ClusterALBInstance settings.
  hostnames:
    - app.example.com
  rules:
    - backendRefs:
        - name: app-svc # Reference to the internal load balancer of the application.
          port: 8080 
---
# Route for HTTPS traffic
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: app-https-route
  namespace: prod
spec:
  parentRefs:
    - name: app-listeners # ListenerSet name.
      namespace: prod
      kind: ListenerSet
      group: gateway.networking.k8s.io
      sectionName: app-https
      port: 443 # HTTPS traffic always uses 443 regardless of ClusterALBInstance settings.
  hostnames:
    - app.example.com
  rules:
    - backendRefs:
        - name: app-svc # Reference to the internal load balancer of the application.
          port: 8080

Publishing an application through an ALBInstance object

In this scenario, the ALBInstance object, the Gateway object, the ListenerSet object, and the HTTPRoute object live in the same namespace.

To publish an application using the ALBInstance object, follow these steps:

  1. Create the ALBInstance object taking into account the required settings:

    apiVersion: network.deckhouse.io/v1alpha1
kind: ALBInstance
metadata:
  name: app-gw
  namespace: prod
spec:
  gatewayName: app-gw
  inlet:
    type: LoadBalancer

  2. After the ALBInstance object reaches the Ready state, create the ListenerSet object and the HTTPRoute object:

    apiVersion: gateway.networking.k8s.io/v1
kind: ListenerSet
metadata:
  name: app-listeners
  namespace: prod
spec:
  parentRef:
    name: app-gw # The name of the Gateway object from the ALBInstance status.
    namespace: prod
  listeners:
    - name: app-https
      port: 443
      protocol: HTTPS
      hostname: app.example.com
      tls:
        mode: Terminate
        certificateRefs:
          - name: app-tls   # Reference to the Secret with the TLS certificate.
            namespace: prod
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: app-route
  namespace: prod
spec:
  parentRefs:
    - name: app-listeners # ListenerSet name.
      namespace: prod
      kind: ListenerSet
      group: gateway.networking.k8s.io
      sectionName: app-https
      port: 443
  hostnames:
    - app.example.com
  rules:
    - backendRefs:
        - name: app-svc # Reference to the internal load balancer of the application.
          port: 8080

GRPCRoute, TLSRoute, and TCPRoute objects

The GRPCRoute object is intended for gRPC traffic. For it, create the ListenerSet object with an HTTPS listener, then add the GRPCRoute object:

apiVersion: gateway.networking.k8s.io/v1
kind: ListenerSet
metadata:
  name: grpc-listeners
  namespace: prod
spec:
  parentRef:
    name: app-gw # The name of the Gateway object from the ALBInstance status.
    namespace: prod
  listeners:
    - name: grpc-https
      port: 443
      protocol: HTTPS
      hostname: grpc.example.com
      tls:
        mode: Terminate
        certificateRefs:
          - name: grpc-tls  # Reference to the Secret with the TLS certificate.
            namespace: prod
---
apiVersion: gateway.networking.k8s.io/v1
kind: GRPCRoute
metadata:
  name: grpc-route
  namespace: prod
spec:
  parentRefs:
    - name: grpc-listeners # ListenerSet name.
      namespace: prod
      kind: ListenerSet
      group: gateway.networking.k8s.io
      sectionName: grpc-https
      port: 443
  hostnames:
    - grpc.example.com
  rules:
    - backendRefs:
        - name: grpc-svc # Reference to the Secret with the TLS certificate.
          port: 9090

For TLS passthrough, when traffic must be decrypted on the application side, either a TLS listener or an HTTPS listener can be used. The example below shows the TLS listener variant:

To accept TCP traffic on an additional port, configure the additionalPorts parameter in the ALBInstance:

apiVersion: network.deckhouse.io/v1alpha1
kind: ALBInstance
metadata:
  name: app-gw
  namespace: prod
spec:
  gatewayName: app-gw
  inlet:
    type: LoadBalancer
    additionalPorts:
      - port: 8443    # An additional TCP port to accept TLS traffic.
        protocol: TCP

Next, configure ListenerSet and TLSRoute objects respectively:

apiVersion: gateway.networking.k8s.io/v1
kind: ListenerSet
metadata:
  name: tls-pass-listeners
  namespace: prod
spec:
  parentRef:
    name: app-gw # The name of the Gateway object from the ALBInstance status.
    namespace: prod
  listeners:
    - name: tls-pass
      port: 8443           # In this case 8443 port is used for TLS.
      protocol: TLS
      hostname: pass.example.com
      tls:
        mode: Passthrough  # TLS passthrough mode is set explicitly.
---
apiVersion: gateway.networking.k8s.io/v1alpha3
kind: TLSRoute
metadata:
  name: tls-pass-route
  namespace: prod
spec:
  parentRefs:
    - name: tls-pass-listeners # ListenerSet name.
      namespace: prod
      kind: ListenerSet
      group: gateway.networking.k8s.io
      sectionName: tls-pass
      port: 8443           # In this case 8443 port is used for TLS.
  hostnames:
    - pass.example.com
  rules:
    - backendRefs:
        - name: tls-pass-svc  # Reference to the internal load balancer of the application.
          port: 8443

The same scenario can also be implemented through an HTTPS listener. This variant is especially useful when the standard handler on port 443 should be used, because no extra port needs to be opened for TLS passthrough:

apiVersion: gateway.networking.k8s.io/v1
kind: ListenerSet
metadata:
  name: https-pass-listeners
  namespace: prod
spec:
  parentRef:
    name: app-gw # The name of the Gateway object from the ALBInstance status.
    namespace: prod
  listeners:
    - name: https-pass
      port: 443 # In this case 443 (HTTPS) port is reused for TLS.
      protocol: HTTPS
      hostname: pass.example.com
      tls:
        mode: Passthrough  # TLS passthrough mode is set explicitly.
---
apiVersion: gateway.networking.k8s.io/v1alpha3
kind: TLSRoute
metadata:
  name: https-pass-route
  namespace: prod
spec:
  parentRefs:
    - name: https-pass-listeners # ListenerSet name.
      namespace: prod
      kind: ListenerSet
      group: gateway.networking.k8s.io
      sectionName: https-pass
      port: 443 # In this case 443 (HTTPS) port is reused for TLS.
  hostnames:
    - pass.example.com
  rules:
    - backendRefs:
        - name: tls-pass-svc # Reference to the internal load balancer of the application.
          port: 8443

If TLS must be terminated on the gateway and then the traffic must be passed further as a regular TCP stream, create a ListenerSet object with a TLS listener in Terminate mode, then attach a TCPRoute object:

apiVersion: gateway.networking.k8s.io/v1
kind: ListenerSet
metadata:
  name: tls-term-listeners
  namespace: prod
spec:
  parentRef:
    name: app-gw # The name of the Gateway object from the ALBInstance status.
    namespace: prod
  listeners:
    - name: tls-term
      port: 443 # In this case 443 (HTTPS) port is reused for TLS.
      protocol: TLS
      hostname: term.example.com
      tls:
        mode: Terminate
        certificateRefs:
          - name: term-tls  # Reference to the Secret with the TLS certificate.
            namespace: prod
---
apiVersion: gateway.networking.k8s.io/v1alpha2
kind: TCPRoute
metadata:
  name: tls-term-route
  namespace: prod
spec:
  parentRefs:
    - name: tls-term-listeners # ListenerSet name.
      namespace: prod
      kind: ListenerSet
      group: gateway.networking.k8s.io
      sectionName: tls-term
      port: 443 # In this case 443 (HTTPS) port is reused for TLS.
  rules:
    - backendRefs:
        - name: tcp-svc # Reference to the internal load balancer of the application.
          port: 8080

Publishing the app via a different gateway

If an application needs to move to another managed Gateway object, change the route attachment in stages:

  1. Create a new ClusterALBInstance object or ALBInstance object so that the controller creates a new Gateway object.
  2. Create a ListenerSet object with the same hostnames, ports, and TLS settings. The new Gateway object must be specified in spec.parentRef.
  3. Add one more parentRefs entry to the existing HTTPRoute object, pointing to the new ListenerSet object.
  4. Verify traffic through the new gateway path.
  5. After verification, remove the reference to the obsolete ListenerSet object from parentRefs of the HTTPRoute object.

Linking routes in one namespace to ListenerSet object in another

If an HTTPRoute object is created in one namespace and must be attached to a ListenerSet object in another namespace, add a ReferenceGrant object in the namespace of the target ListenerSet object. The example below shows a shared ListenerSet object in namespace shared-gw, an application HTTPRoute object in namespace prod, and a ReferenceGrant object that allows this attachment:

apiVersion: gateway.networking.k8s.io/v1
kind: ListenerSet
metadata:
  name: shared-listeners
  namespace: shared-gw
spec:
  parentRef:
    name: public-gw
    namespace: d8-alb
  listeners:
    - name: app-https
      port: 443
      protocol: HTTPS
      hostname: app.example.com
      tls:
        mode: Terminate
        certificateRefs:
          - name: app-tls
            namespace: shared-gw
---
apiVersion: gateway.networking.k8s.io/v1
kind: ReferenceGrant
metadata:
  name: allow-prod-httproute-to-shared-listeners
  namespace: shared-gw
spec:
  from:
    - group: gateway.networking.k8s.io
      kind: HTTPRoute
      namespace: prod
  to:
    - group: gateway.networking.k8s.io
      kind: ListenerSet
      name: shared-listeners
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: app-route
  namespace: prod
spec:
  parentRefs:
    - name: shared-listeners
      namespace: shared-gw
      kind: ListenerSet
      group: gateway.networking.k8s.io
      sectionName: app-https
      port: 443
  hostnames:
    - app.example.com
  rules:
    - backendRefs:
        - name: app-svc
          port: 8080

Configuring TLS parameters with BackendTLSPolicy

If traffic from the gateway to the backend must use TLS, create a BackendTLSPolicy object in the namespace of the backend Service object. The example below shows an HTTPRoute object, a backend Service object with a named port, a ConfigMap with a CA bundle, and a BackendTLSPolicy object that configures TLS validation for that backend:

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: app-route
  namespace: prod
spec:
  parentRefs:
    - name: app-listeners
      namespace: prod
      kind: ListenerSet
      group: gateway.networking.k8s.io
      sectionName: app-https
      port: 443
  hostnames:
    - app.example.com
  rules:
    - backendRefs:
        - name: app-svc
          port: 8443
---
apiVersion: v1
kind: Service
metadata:
  name: app-svc
  namespace: prod
spec:
  selector:
    app: app
  ports:
    - name: https
      port: 8443
      targetPort: 8443
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: app-backend-ca
  namespace: prod
data:
  ca.crt: |
    -----BEGIN CERTIFICATE-----
    ...
    -----END CERTIFICATE-----
---
apiVersion: gateway.networking.k8s.io/v1
kind: BackendTLSPolicy
metadata:
  name: app-svc-tls
  namespace: prod
spec:
  targetRefs:
    - group: ""
      kind: Service
      name: app-svc
      sectionName: https
  validation:
    hostname: app.internal.example.com
    caCertificateRefs:
      - group: ""
        kind: ConfigMap
        name: app-backend-ca

Configuring OpenTelemetry tracing TLS

If OpenTelemetry tracing must send data over TLS, create a Kubernetes Secret with the CA certificate and reference it from spec.openTelemetry.tracing.tls.caSecretName.

For ClusterALBInstance and the default deckhouse gateway, place the Secret in the d8-alb namespace. The Secret must contain the cacert key.

apiVersion: v1
kind: Secret
metadata:
  name: otel-tracing-ca
  namespace: d8-alb
type: Opaque
stringData:
  cacert: |
    -----BEGIN CERTIFICATE-----
    ...
    -----END CERTIFICATE-----
---
apiVersion: network.deckhouse.io/v1alpha1
kind: ClusterALBInstance
metadata:
  name: proxy-gw
spec:
  gatewayName: proxy-gw
  openTelemetry:
    tracing:
      service:
        name: otel-collector
        namespace: monitoring
      port: 4318
      protocol: HTTP
      path: /v1/traces
      tls:
        sni: otel-collector.monitoring.svc.cluster.local
        caSecretName: otel-tracing-ca

Supported HTTPRoute annotations

Because the current Gateway API specification does not yet cover all features required for a Deckhouse cluster to operate properly, the module provides a gradually growing set of HTTPRoute object annotations that adds the missing configuration options. The controller reads these keys from HTTPRoute.metadata.annotations.

Annotation Description
alb.network.deckhouse.io/tls-disable-protocol Disables a TLS protocol version for the handler with the hostname of this route (for example value http2). This may be required in rare cases when a shared certificate with several DNS names is used together with request redirection.
alb.network.deckhouse.io/whitelist-source-range Expects a comma-separated list of subnets in CIDR format: an IP filter at route level; overrides the global whitelist (for example 10.1.1.10/32, 10.2.2.2/32).
alb.network.deckhouse.io/response-headers-to-add JSON object with additional response headers (for example {"Strict-Transport-Security": "max-age=31536000; includeSubDomains"}).
alb.network.deckhouse.io/session-affinity JSON for cookie session affinity (mode, path, cookieName, ttl, etc.); not every field is required (for example {"mode": "cookie", "path": "/path", "cookieName": "mycookie", "ttl": 0}).
alb.network.deckhouse.io/hash-key For example source-ip: consistent hashing for Service backends of the HTTPRoute object.
alb.network.deckhouse.io/service-upstream "true": traffic to the upstream goes through the corresponding Service object instead of directly to pods.
alb.network.deckhouse.io/basic-auth-secret namespace/secret with htpasswd data for HTTP basic auth on this route.
alb.network.deckhouse.io/satisfy all or any: defines whether both checks must be satisfied (whitelist and basic-auth) or only one of them (default all).
alb.network.deckhouse.io/auth-url Defines the URL of the external authentication service.
alb.network.deckhouse.io/auth-signin Defines the redirect URL for authentication when 401 is returned by external authentication.
alb.network.deckhouse.io/auth-response-headers Comma-separated list: additional headers from the auth response to pass upstream (on top of the standard allowlist).
alb.network.deckhouse.io/mod-security JSON configuration for the per-route ModSecurity/Coraza WAF.
alb.network.deckhouse.io/rewrite-target Allows rewriting paths for rules with RegularExpression type by using regex capture groups (for example /my-path/\1).
alb.network.deckhouse.io/buffer-max-request-bytes Defines the buffer size that may be used when requests are buffered (by default Envoy Proxy does not buffer requests).
alb.network.deckhouse.io/limit-rps RPS limit for a route.
alb.network.deckhouse.io/backend-tls-settings For example {"mode": "SIMPLE", "insecureSkipVerify": true, "clientCertificate": "", "privateKey": "", "caCertificates": ""}; allows explicit configuration of TLS connection parameters to the upstream.

WAF on HTTPRoute

The alb.network.deckhouse.io/mod-security annotation enables the ModSecurity/Coraza WAF for a specific HTTPRoute. The WAF is configured per route and does not affect other routes unless the same annotation is added there.

Supported annotation fields:

Field Description
mode WAF engine mode: on, off, or any other value for DetectionOnly.
preset Optional preset ruleset. Currently only owasp-crs is supported. If the field is omitted, no preset rules are loaded.
paranoiaLevel Optional CRS paranoia level from 1 to 4. Applied only when preset is owasp-crs.
configRef.namespace Optional namespace of the ConfigMap with custom rules. Defaults to the namespace of the HTTPRoute.
configRef.name Name of the ConfigMap with custom rules.
configRef.key Optional key in the ConfigMap. If omitted, all keys are read in sorted order.
directives Optional inline list of ModSecurity/Coraza directives appended after preset and ConfigMap rules.

Directive order:

  1. Base directives shipped with the module (@coraza.conf, SecRuleEngine, SecResponseBodyAccess Off).
  2. Preset rules from preset.
  3. Rules from configRef.
  4. Inline directives.

Inline directives are applied last, so they can override preset or ConfigMap rules.

Minimal example:

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: app
  namespace: prod
  annotations:
    alb.network.deckhouse.io/mod-security: |
      {
        "mode": "on"
      }
spec:
  hostnames:
    - app.example.com
  parentRefs:
    - group: gateway.networking.k8s.io
      kind: ListenerSet
      name: app-listeners
      namespace: prod
      sectionName: app-https
      port: 443
  rules:
    - matches:
        - path:
            type: PathPrefix
            value: /
      backendRefs:
        - name: app-svc
          port: 8080

Example with the OWASP CRS preset:

metadata:
  annotations:
    alb.network.deckhouse.io/mod-security: |
      {
        "mode": "on",
        "preset": "owasp-crs",
        "paranoiaLevel": 1
      }

Example with custom rules from a ConfigMap:

apiVersion: v1
kind: ConfigMap
metadata:
  name: waf-rules
  namespace: prod
data:
  rules.conf: |
    SecRule ARGS:test "@streq block" \
      "id:1000001,phase:2,deny,status:403,msg:'test waf block'"
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: app
  namespace: prod
  annotations:
    alb.network.deckhouse.io/mod-security: |
      {
        "mode": "on",
        "preset": "owasp-crs",
        "paranoiaLevel": 1,
        "configRef": {
          "name": "waf-rules",
          "key": "rules.conf"
        },
        "directives": [
          "SecResponseBodyAccess Off"
        ]
      }

Rule syntax reference:

Current notes and limitations:

  • only the owasp-crs preset is supported
  • paranoiaLevel is ignored when preset is omitted or differs from owasp-crs
  • valid paranoiaLevel values are 1..4; in practice it is recommended to start with 1
  • the WAF currently inspects only incoming requests to the application and can block such requests when rules match; responses sent back to the client are not inspected
  • rules from ConfigMap values may be multiline: lines ending with \ are joined automatically.

Using GeoIP and GeoLite2

The alb module supports enriching incoming HTTP requests with headers based on MaxMind GeoIP/GeoLite2 databases.

Currently, the following database editions can be used:

  • GeoIP2-Anonymous-IP;
  • GeoIP2-City;
  • GeoIP2-ISP;
  • GeoIP2-ASN;
  • GeoLite2-ASN;
  • GeoLite2-City.

The current GeoIP integration supports using up to 4 databases simultaneously.

Downloading GeoIP Databases from MaxMind

To use GeoIP and download databases directly from MaxMind servers, you first need to create a secret containing the license key, for example:

d8 k -n prod create secret generic geoip-license --from-literal=licenseKey='<MAXMIND_LICENSE_KEY>'

When configuring GeoIP for ClusterALBInstance, the secret can be placed in any namespace, but it is recommended to use d8-alb.

For ALBInstance objects, the secret must reside in the same namespace as the ALBInstance object.

After creating the secret, you need to reference it in a ClusterALBInstance or ALBInstance object, for example:

apiVersion: network.deckhouse.io/v1alpha1
kind: ALBInstance
metadata:
  name: main
  namespace: prod
spec:
  envoyLogLevel: Warning
  gatewayName: custom-gateway
  geoIP:
    licenseKeySecretRef:
      name: geoip-license

Downloading GeoIP Databases from a Local Mirror

To use GeoIP and download databases from a local mirror, specify the mirror URL, for example:

apiVersion: network.deckhouse.io/v1alpha1
kind: ALBInstance
metadata:
  name: main
  namespace: prod
spec:
  envoyLogLevel: Warning
  gatewayName: custom-gateway
  geoIP:
    maxmindMirror:
      url: "https://local.geoip:8443"

You can also use a URL pointing to a local caching GeoIP server in another namespace, for example:

apiVersion: network.deckhouse.io/v1alpha1
kind: ALBInstance
metadata:
  name: main
  namespace: prod
spec:
  envoyLogLevel: Warning
  gatewayName: custom-gateway
  geoIP:
    maxmindMirror:
      url: "http://geoproxy-cluster.d8-alb.svc:8080/download"

Using GeoIP Headers

Once GeoIP is configured in the namespace where the ClusterALBInstance or ALBInstance resides, a caching and update server for GeoIP databases will be started, and Envoy Proxy pods will be sequentially restarted with functionality to fetch GeoIP databases from the local GeoIP server.

To enrich HTTP requests with GeoIP-based data, specify the names of the HTTP headers that will contain the corresponding information, for example:

apiVersion: network.deckhouse.io/v1alpha1
kind: ALBInstance
metadata:
  name: main
  namespace: prod
spec:
  envoyLogLevel: Warning
  gatewayName: custom-gateway
  geoIP:
    headers:
      city: geoip_city
      country: geoip_country
    licenseKeySecretRef:
      name: geoip-license
    maxmindEditionIDs:
      - GeoLite2-City

GeoIP databases are updated once per day, both on the caching server and in each individual Envoy Proxy pod using the caching server.

OpenTelemetry Tracing Configuration

The alb module supports exporting OpenTelemetry traces from Envoy proxies.

To enable export, specify the target OpenTelemetry Collector address as a URL. Traces can be transmitted over OTLP/HTTP or OTLP/gRPC. TLS can be optionally configured for secure connections.

When using TLS, it is recommended to explicitly set the SNI parameter if the OpenTelemetry Collector is behind a proxy or load balancer that selects upstreams based on Server Name Indication.