The ingress-nginx module: examples

An example for AWS (Network Load Balancer)

When creating a balancer, all zones available in the cluster are used.

In each zone, the balancer receives a public IP. If there is an instance with an Ingress controller in the zone, an A-record with the balancer’s IP address from this zone is automatically added to the balancer’s domain name.

When there are no instances with an Ingress controller in the zone, then the IP is automatically removed from the DNS.

If there is only one instance with an Ingress controller in a zone, when the pod is restarted, the IP address of the balancer of this zone will be temporarily excluded from DNS.

apiVersion: deckhouse.io/v2
kind: IngressNginxController
metadata:
 name: main
spec:
  ingressClass: "nginx"
  inlet: "LoadBalancer"
  loadBalancer:
    annotations:
      service.beta.kubernetes.io/aws-load-balancer-type: "nlb"

An example for GCP / Yandex Cloud / Azure

apiVersion: deckhouse.io/v2
kind: IngressNginxController
metadata:
 name: main
spec:
  ingressClass: "nginx"
  inlet: "LoadBalancer"

An example for OpenStack

apiVersion: deckhouse.io/v2
kind: IngressNginxController
metadata:
  name: main-lbwpp
spec:
  inlet: LoadBalancerWithProxyProtocol
  ingressClass: nginx
  loadBalancerWithProxyProtocol:
    annotations:
      loadbalancer.openstack.org/proxy-protocol: "true"
      loadbalancer.openstack.org/timeout-member-connect: "2000"

An example for Bare metal

apiVersion: deckhouse.io/v2
kind: IngressNginxController
metadata:
  name: main
spec:
  ingressClass: nginx
  inlet: HostWithFailover
  nodeSelector:
    node-role.deckhouse.io/frontend: ""
  tolerations:
  - effect: NoExecute
    key: dedicated.deckhouse.io
    value: frontend

An example for Bare metal (Behind external load balancer, e.g. Cloudflare, Qrator, Nginx+, Citrix ADC, Kemp, etc.)

apiVersion: deckhouse.io/v2
kind: IngressNginxController
metadata:
  name: main
spec:
  ingressClass: nginx
  inlet: HostPort
  hostPort:
    httpPort: 80
    httpsPort: 443
    behindL7Proxy: true

An example for Bare metal (MetalLB BGP LoadBalancer)

apiVersion: deckhouse.io/v2
kind: IngressNginxController
metadata:
  name: main
spec:
  ingressClass: nginx
  inlet: LoadBalancer
  nodeSelector:
    node-role.deckhouse.io/frontend: ""
  tolerations:
  - effect: NoExecute
    key: dedicated.deckhouse.io
    value: frontend

In the case of using MetalLB, its speaker Pods must be run on the same Nodes as the Ingress controller Pods.

The controller must receive real IP addresses of clients — therefore its Service is created with the parameter externalTrafficPolicy: Local (disabling cross–node SNAT), and to accept this parameter the MetalLB speaker announce this Service only from those Nodes where the target Pods are running.

For this example, the configuration of the metallb module should be as follows:

metallb:
 speaker:
   nodeSelector:
     node-role.deckhouse.io/frontend: ""
   tolerations:
    - effect: NoExecute
      key: dedicated.deckhouse.io
      value: frontend

An example for Bare metal (MetalLB L2 LoadBalancer)

1. Enable the metallb module:

   apiVersion: deckhouse.io/v1alpha1
   kind: ModuleConfig
   metadata:
     name: metallb
   spec:
     enabled: true
     version: 2

2. Create a MetalLoadBalancerClass resource:

   Metallb balancers should be placed on the same nodes as ingress controllers. In typical deployment scenarios, frontend nodes are used for this purpose (to deploy ingress controllers and Metallb load balancers on frontend nodes, use the annotation node-role.deckhouse.io/frontend: "" in their manifests).

   apiVersion: network.deckhouse.io/v1alpha1
   kind: MetalLoadBalancerClass
   metadata:
     name: ingress
   spec:
     addressPool:
       - 192.168.2.100-192.168.2.150
     isDefault: false
     nodeSelector:
       node-role.deckhouse.io/frontend: ""
     type: L2

3. Create a IngressNginxController resource:

   apiVersion: deckhouse.io/v2
   kind: IngressNginxController
   metadata:
     name: main
   spec:
     ingressClass: nginx
     inlet: LoadBalancer
     loadBalancer:
       loadBalancerClass: ingress
       annotations:
       # The number of addresses that will be allocated from the pool declared in MetalLoadBalancerClass.
       network.deckhouse.io/l2-load-balancer-external-ips-count: "3"
     # Label selector and tolerations. Ingress-controller pods must be scheduled on same nodes as MetalLB speaker pods.
     nodeSelector:
       node-role.deckhouse.io/frontend: ""
     tolerations:
     - effect: NoExecute
       key: dedicated.deckhouse.io
       value: frontend
       operator: Equal

   When creating an ingress controller, you can also specify certain IP addresses from the pool that will be assigned to it. To specify the addresses that should be assigned to the service, use the annotation network.deckhouse.io/load-balancer-ips. If there is more than one desired address, there must also be an annotation network.deckhouse.io/l2-load-balancer-external-ips-count, which must specify the number of addresses allocated from the pool (it must not be less than the number of addresses listed in network.deckhouse.io/load-balancer-ips). Example of using annotations to assign specific addresses from the pool to the service.

4. The platform will create a service with the type LoadBalancer, to which a specified number of addresses will be assigned:

   d8 k -n d8-ingress-nginx get svc
   NAME                   TYPE           CLUSTER-IP      EXTERNAL-IP                                 PORT(S)                      AGE
   main-load-balancer     LoadBalancer   10.222.130.11   192.168.2.100,192.168.2.101,192.168.2.102   80:30689/TCP,443:30668/TCP   11s

Example of segregating access between public and administrative zones

In many applications, the same backend serves both the public part and the administrative interface. For example:

• https://example.com is the public zone;
• https://admin.example.com is the administrative zone, access to which must be restricted (ACL, mTLS, IP whitelist, and so on).

For this scenario, we recommend offloading administrative traffic to a separate Ingress controller (with a dedicated Ingress class if necessary) and restricting access to it by using the spec.acceptRequestsFrom parameter.

Specifics of using a single Ingress controller

Consider an example where a single Ingress controller is used to serve requests from both the public zone and the administrative interface.

Example of Ingress resource configuration for this case:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: admin-ingress
  annotations:
    nginx.ingress.kubernetes.io/whitelist-source-range: "1.2.3.4/32"
spec:
  ingressClassName: nginx # The Ingress resource for administrative traffic is associated with the same Ingress controller as the Ingress resource for public traffic.
  rules:
    - host: admin.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: backend
                port:
                  number: 80
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: public-ingress
spec:
  ingressClassName: nginx # The Ingress resource for public traffic is associated with the same Ingress controller as the Ingress resource for administrative traffic.
  rules:
    - host: example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: backend
                port:
                  number: 80

With processing and forwarding of X-Forwarded-* headers enabled, the backend can rely on the x-forwarded-host header when making authorization decisions. In the example above, it is possible to reach the administrative zone through the Ingress resource that serves public traffic by using x-forwarded-host. Therefore, when using this option you must be sure that requests to the Ingress controller come only from trusted sources.

Using separate Ingress controllers

To avoid the situation described above (when, with processing and forwarding of X-Forwarded-* headers enabled it is possible, for example, to reach the administrative zone via the Ingress resource that serves public traffic by using x-forwarded-host), we recommend that you:

• configure access rules at the Ingress resource level,
• use separate Ingress controllers,
• restrict which source addresses are allowed to connect to the Ingress controllers.

Example of Ingress resource configuration for this case:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: admin-ingress
  annotations:
    nginx.ingress.kubernetes.io/whitelist-source-range: "1.2.3.4/32"
spec:
  ingressClassName: admin-nginx # The Ingress resource for administrative traffic is associated with a separate Ingress controller.
  rules:
    - host: admin.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: backend
                port:
                  number: 80
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: public-ingress
spec:
  ingressClassName: public-nginx # The Ingress resource for public traffic is associated with a separate Ingress controller.
  rules:
    - host: example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: backend
                port:
                  number: 80

Example of an Ingress controller that serves administrative Ingress resources and accepts connections only from specified subnets:

apiVersion: deckhouse.io/v2
kind: IngressNginxController
metadata:
  name: admin
spec:
  ingressClass: admin-nginx
  inlet: HostPort
  acceptRequestsFrom:
    - 1.2.3.4/32
    - 10.0.0.0/16
  hostPort:
    httpPort: 80
    httpsPort: 443
    behindL7Proxy: true

In this example:

• The Ingress controller is exposed on node ports through the HostPort inlet.
• The acceptRequestsFrom parameter allows connections to the controller only from the listed subnets.
• Even if an external load balancer or client can set its own X-Forwarded-* header values, the decision whether to allow the connection to reach the controller is made based on the actual source address, not on headers.
• Administrative Ingress resources (in this example admin-ingress) are served by this controller according to the configured Ingress class.

Example of an Ingress controller that serves Ingress resources for public traffic:

apiVersion: deckhouse.io/v2
kind: IngressNginxController
metadata:
  name: public
spec:
  ingressClass: public-nginx
  inlet: HostPort
  hostPort:
    httpPort: 8080
    httpsPort: 8443
    behindL7Proxy: true

An example of using cert-manager to issue certificates

The module works with certificates issued by cert-manager. The Ingress controller uses the Secret referenced in the spec.tls section of the Ingress resource, so the main requirement is that cert-manager creates this Secret in the Ingress namespace.

The examples below use a ClusterIssuer with an HTTP-01 solver. Create the ClusterIssuer (or a namespaced Issuer) according to the cert-manager module documentation:

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt
spec:
  acme:
    email: admin@example.com
    server: https://acme-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      name: letsencrypt-account-key
    solvers:
      - http01:
          ingress:
            ingressClassName: nginx

Replace the email address and, if necessary, the ACME server. Use the staging ACME server while testing to avoid Let’s Encrypt rate limits.

For a namespaced Issuer, use the same spec with kind: Issuer and set metadata.namespace; then use the cert-manager.io/issuer annotation or issuerRef.kind: Issuer as shown below.

The ingressClassName in the solver must match the Ingress class of the controller that serves the domain. In the examples below it is nginx. If your controller uses another class, change it in both the solver and the Ingress resource.

Ordering a certificate with an Ingress annotation

Add the cert-manager.io/cluster-issuer annotation to the Ingress resource and specify its spec.tls section:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: my-app
  namespace: my-namespace
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt
spec:
  ingressClassName: nginx
  tls:
    - hosts:
        - my-app.example.com
      secretName: my-app-tls
  rules:
    - host: my-app.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: my-app
                port:
                  number: 80

cert-manager creates a Certificate resource from the annotation and the spec.tls section, issues the certificate, and writes it to the my-app-tls Secret. The hosts in spec.tls are the DNS names cert-manager orders; they must match the hosts in the Ingress resource rules. The Ingress controller picks up the Secret automatically and starts serving HTTPS.

Use cert-manager.io/issuer: <issuer-name> instead of cert-manager.io/cluster-issuer to order a certificate from a namespaced Issuer. In this case, the Issuer and the Ingress resource must be in the same namespace.

Ordering a certificate with a dedicated Certificate resource

Create a Certificate resource that references the issuer (Issuer or ClusterIssuer) and the target Secret:

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: my-app-tls
  namespace: my-namespace
spec:
  secretName: my-app-tls
  issuerRef:
    name: letsencrypt
    kind: ClusterIssuer
    group: cert-manager.io
  dnsNames:
    - my-app.example.com

Then reference the same Secret in the Ingress resource. No cert-manager annotation is needed on the Ingress resource:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: my-app
  namespace: my-namespace
spec:
  ingressClassName: nginx
  tls:
    - hosts:
        - my-app.example.com
      secretName: my-app-tls
  rules:
    - host: my-app.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: my-app
                port:
                  number: 80

This method is convenient when the certificate is used by several Ingress resources in the same namespace or managed separately from them. The Certificate resource and the Ingress resource must be in the same namespace, because spec.tls.secretName refers to a Secret in the namespace of the Ingress resource. For a namespaced Issuer, set issuerRef.kind: Issuer; the Issuer must be in the same namespace as the Certificate.

In both cases, cert-manager creates a Secret that the Ingress controller watches. When cert-manager renews the certificate, it updates the Secret, and the controller reloads the certificate automatically.

The module-level defaultSSLCertificate is a separate fallback certificate and is not required for cert-manager.