The ingress-nginx module: FAQ

How do I limit access to the application in the cluster to Ingress controllers only?

If you need to limit access to your application inside the cluster exclusively from Ingress Pods, you should add a kube-rbac-proxy container to the application Pod as shown in the example below:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
  namespace: my-namespace
spec:
  selector:
    matchLabels:
      app: my-app
  replicas: 1
  template:
    metadata:
      labels:
        app: my-app
    spec:
      serviceAccountName: my-sa
      containers:
      - name: my-cool-app
        image: mycompany/my-app:v0.5.3
        args:
        - "--listen=127.0.0.1:8080"
        livenessProbe:
          httpGet:
            path: /healthz
            port: 443
            scheme: HTTPS
      - name: kube-rbac-proxy
        image: flant/kube-rbac-proxy:v0.1.0
        # It is recommended to use a proxy from Deckhouse repository.
        args:
        - "--secure-listen-address=0.0.0.0:443"
        - "--config-file=/etc/kube-rbac-proxy/config-file.yaml"
        - "--v=2"
        - "--logtostderr=true"
        # The user authentication and authorization are not possible if the kube-apiserver is not available.
        # Stale Cache stores the results of successful authorization and is used only if the apiserver is not available.
        - "--stale-cache-interval=1h30m"
        ports:
        - containerPort: 443
          name: https
        volumeMounts:
        - name: kube-rbac-proxy
          mountPath: /etc/kube-rbac-proxy
      volumes:
      - name: kube-rbac-proxy
        configMap:
          name: kube-rbac-proxy

The application only accepts localhost (127.0.0.1) requests. That means that an unsecured connection can only be established to it from within the Pod. At the same time, the proxy listens on 0.0.0.0 and intercepts all external traffic to the Pod.

How do I provide minimum rights to the Service Account?

The proxy needs permissions to create TokenReview and SubjectAccessReview to authenticate and authorize users using the kube-apiserver.

DKP clusters already have a ready-made ClusterRole — d8-rbac-proxy, you don’t need to create it yourself! Link it to your Deployment’s ServiceAccount, as shown in the example below.

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: my-sa
  namespace: my-namespace
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: my-namespace:my-sa:d8-rbac-proxy
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: d8:rbac-proxy
subjects:
- kind: ServiceAccount
  name: my-sa
  namespace: my-namespace

The Kube-RBAC-Proxy configuration

apiVersion: v1
kind: ConfigMap
metadata:
  name: kube-rbac-proxy
data:
  config-file.yaml: |+
    excludePaths:
    - /healthz
  # No authorization for liveness probes is required.
    upstreams:
    - upstream: http://127.0.0.1:8081/
  # The destination address
      path: /
  # The path to the proxy to forward requests to the upstream.
      authorization:
        resourceAttributes:
          namespace: my-namespace
          apiGroup: apps
          apiVersion: v1
          resource: deployments
          subresource: http
          name: my-app

According to the configuration, the user must have access to the my-app Deployment and its http subresource in the my-namespace namespace.

Such permissions have the following RBAC form:

---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: kube-rbac-proxy:my-app
  namespace: my-namespace
rules:
- apiGroups: ["apps"]
  resources: ["deployments/http"]
  resourceNames: ["my-app"]
  verbs: ["get", "create", "update", "patch", "delete"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: kube-rbac-proxy:my-app
  namespace: my-namespace
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: kube-rbac-proxy:my-app
subjects:
# All user certificates of Ingress controllers are issued for one specific group.
- kind: Group
  name: ingress-nginx:auth

For the Ingress resource, add parameters:

nginx.ingress.kubernetes.io/backend-protocol: HTTPS
nginx.ingress.kubernetes.io/configuration-snippet: |
  proxy_ssl_certificate /etc/nginx/ssl/client.crt;
  proxy_ssl_certificate_key /etc/nginx/ssl/client.key;
  proxy_ssl_protocols TLSv1.2;
  proxy_ssl_session_reuse on;

You can read more about X.509 client certificate authentication in the Kubernetes documentation.

How do I configure an external load balancer to check if IngressNginxController is available?

In case an Ingress NGINX Controller is deployed behind a load balancer, it is recommended that you configure your load balancer so that it would check the availability of the Ingress NGINX Controller’s endpoints via a health check mechanism, periodically sending either HTTP requests or TCP packets. While it is possible to test the endpoints simply by checking if a relevant TCP port is open, we recommend implementing HTTP checks with the following parameters:

• Protocol: HTTP
• Path: /healthz
• Port: 80 (or relevant httpPort value in case of using HostPort inlet).

How do I configure MetalLB to be accessible from the internal network only?

Below is an example of a MetalLB config with access from the internal network only:

apiVersion: deckhouse.io/v1
kind: IngressNginxController
metadata:
  name: main
spec:
  ingressClass: "nginx"
  inlet: "LoadBalancer"
  loadBalancer:
    sourceRanges:
    - 192.168.0.0/24

How to add extra log fields to a nginx-controller?

Example of adding extra fields:

apiVersion: deckhouse.io/v1
kind: IngressNginxController
metadata:
  name: main
spec:
  ingressClass: "nginx"
  inlet: "LoadBalancer"
  additionalLogFields:
    my-cookie: "$cookie_MY_COOKIE"

How to enable HorizontalPodAutoscaling for IngressNginxController?

HPA is set with attributes minReplicas and maxReplicas in a IngressNginxController CR.

The IngressNginxController is deployed using DaemonSet. DaemonSet does not provide horizontal scaling capabilities, so hpa-scaler Deployment will be created with the HPA resource, which is observing custom metric prometheus-metrics-adapter-d8-ingress-nginx-cpu-utilization-for-hpa. If CPU utilization exceeds 50%, the HPA-controller scales hpa-scaler Deployment with a new replica (with respect to minReplicas and maxReplicas).

hpa-scaler Deployment has HardPodAntiAffinity, and it will order a new Node (inside its NodeGroup), where one more ingress-controller will be set.

Notes:

• The minimum actual number of IngressNginxController replicas cannot be less than the minimum number of nodes in the node group it is deployed to.
• The maximum actual number of IngressNginxController replicas cannot be greater than the maximum number of nodes in the node group it is deployed to.

How to use IngressClass with IngressClassParameters?

Since version 1.1 IngressNginxController Deckhouse creates an IngressClass object. If you want to use your own IngressClass with your customized IngressClassParameters, you need to add the label ingress-class.deckhouse.io/external: "true":

apiVersion: networking.k8s.io/v1
kind: IngressClass
metadata:
  labels:
    ingress-class.deckhouse.io/external: "true"
  name: my-super-ingress
spec:
  controller: ingress-nginx.deckhouse.io/my-super-ingress
  parameters:
    apiGroup: elbv2.k8s.aws
    kind: IngressClassParams
    name: awesome-class-cfg

In this case Deckhouse will not create an IngressClass object and will use your own.

How to disable the collection of detailed Ingress resources statistics?

By default, Deckhouse collects detailed statistics from all Ingress resources in the cluster, which generates a high load on the monitoring system.

To disable statistics collection, add label ingress.deckhouse.io/discard-metrics: "true" to the corresponding Namespace or Ingress resource.

Example of disabling statistics (metrics) collection for all Ingress resources in the review-1 namespace:

d8 k label ns review-1 ingress.deckhouse.io/discard-metrics=true

Example of disabling statistics (metrics) collection for all test-site Ingress resources in the development namespace:

d8 k label ingress test-site -n development ingress.deckhouse.io/discard-metrics=true

How do I correctly drain a node running an IngressNginxController’s pods?

There are two ways to gracefully drain a node running IngressNginxController.

1. Using an annotation.

   The annotation will be automatically removed after the operation completes.

   d8 k annotate node <node_name> update.node.deckhouse.io/draining=user

2. Using d8 k drain.

   When using the standard d8 k drain command, you must specify the --force flag even if --ignore-daemonsets is present, since IngressNginxController is deployed using Advanced DaemonSet:

   d8 k drain <node_name> --delete-emptydir-data --ignore-daemonsets --force

How to enable Web Application Firewall (WAF)?

Software known as a Web Application Firewall (WAF) is used to protect web applications from Layer 7 attacks. Ingress-nginx controller has a built-in WAF called `ModSecurity’ (The Open Worldwide Application Security Project).

ModSecurity is disabled by default.

Enabling ModSecurity

To enable ModSecurity, you must specify the following parameters in the config section of the CR IngressNginxController:

apiVersion: deckhouse.io/v1
kind: IngressNginxController
metadata:
  name: <name_of_the_controller>
spec:
  config:
    enable-modsecurity: "true"
    modsecurity-snippet: |
      Include /etc/nginx/modsecurity/modsecurity.conf

After applying the settings, ModSecurity will start working for all traffic passing through this Ingress nginx controller. This uses the audit mode (DetectionOnly) and basic recommended configuration.

Setting up ModSecurity

You can configure ModSecurity in two ways:

1. For the entire ingress-nginx controller
   • the necessary directives are described in the section config.modsecurity-snippet in CR IngressNginxController, as in the example above.
2. For each CR Ingress separately
   • the necessary directives are described in the annotation nginx.ingress.kubernetes.io/modsecurity-snippet : | directly in Ingress manifests.

To enable the execution of rules (and not just logging), add the SecRuleEngine On directive as shown below:

apiVersion: deckhouse.io/v1
kind: IngressNginxController
metadata:
  name: <name_of_the_controller>
spec:
  config:
    enable-modsecurity: "true"
    modsecurity-snippet: |
      Include /etc/nginx/modsecurity/modsecurity.conf
      SecRuleEngine On

Adding SecRules

An example of a rule to limit the number of arguments in a request URL. If the number of arguments exceeds 10, the server will reject the request with an error code 400 (Bad Request).

apiVersion: deckhouse.io/v1
kind: IngressNginxController
metadata:
  name: <name_of_the_controller>
spec:
  config:
    enable-modsecurity: "true"
    modsecurity-snippet: |
      Include /etc/nginx/modsecurity/modsecurity.conf
      SecRuleEngine On
      SecRule &ARGS "@gt 10" "id:100100,phase:2,deny,status:400,log,auditlog,severity:WARNING,msg:\"too many args\""

A full list and description of the directives can be found in the official documentation.

Enabling OWASP Core Rule Set (CRS)

To enable OWASP Core Rule Set (CRS) add the config.enable-owasp-modsecurity-crs in CR IngressNginxController, as in the example below.

apiVersion: deckhouse.io/v1
kind: IngressNginxController
metadata:
  name: <name_of_the_controller>
spec:
  config:
    enable-modsecurity: "true"
    enable-owasp-modsecurity-crs: "true"
    modsecurity-snippet: |
      Include /etc/nginx/modsecurity/modsecurity.conf
      SecRuleEngine On

Additional Ingress annotations

Why are additional annotations needed?

Snippet annotations (nginx.ingress.kubernetes.io/configuration-snippet, nginx.ingress.kubernetes.io/server-snippet, nginx.ingress.kubernetes.io/auth-snippet) are designed to inject arbitrary directives into the NGINX configuration: their content is inserted verbatim, without any validation. The controller cannot tell a deliberate change from an erroneous or a malicious one, and all of them end up in the single NGINX configuration shared by every Ingress of the controller. A mistake in a snippet of one Ingress can break others, and anyone allowed to create Ingresses can run arbitrary NGINX directives and Lua code: read the controller files, including its ServiceAccount token and the TLS keys of all Ingresses, or send requests to internal cluster services. CVE-2021-25742, for example, exploits exactly this. For these reasons, upstream ingress-nginx disables snippet annotations by default.

In the module, snippets are still allowed by default: unless annotationPolicy says otherwise, the controller runs with allow-snippet-annotations: "true" and annotations-risk-level: Critical. The annotations below cover the most common uses of snippets, so that snippets can be turned off with annotationPolicy.allowSnippetAnnotations: false:

• each value is validated, and an invalid value rejects the Ingress;
• the NGINX configuration is generated from fixed templates, and an annotation value never gets into it as is;
• the annotations work with allowSnippetAnnotations: false.

If snippet annotations have to stay enabled, annotationPolicy.valueWordBlocklist restricts what they may contain: the controller rejects an Ingress if the value of any of its nginx.ingress.kubernetes.io/ annotations contains one of the listed words, for example load_module, lua_package, _by_lua, location, root, proxy_pass or serviceaccount. Such a list filters out known dangerous constructs but does not make snippets safe. It applies to all annotations, including the additional ones below, so it must not contain words their values may need.

The annotations are Deckhouse extensions: upstream ingress-nginx does not have them. They work in every supported controller version.

Risk levels

If annotationPolicy.riskLevel is set, the controller rejects an Ingress that has an annotation of a higher level. Without it, annotations of every level are accepted.

The level in brackets next to each example below is the highest level of all its annotations, upstream ones included, that is, the lowest riskLevel the example works with.

Snippets that need no replacement

Some directives repeat what the controller already does and can simply be removed:

• proxy_set_header Authorization $http_authorization; and proxy_set_header X-Auth-Token $http_x_auth_token; in nginx.ingress.kubernetes.io/auth-snippet: all request headers are passed to the authentication service;
• proxy_set_header X-Real-IP $remote_addr; in nginx.ingress.kubernetes.io/auth-snippet: the controller sets this header itself.

Static headers on every response, such as X-Frame-Options or Content-Security-Policy, are set with the upstream nginx.ingress.kubernetes.io/custom-headers annotation and a ConfigMap. X-Frame-Options, X-Content-Type-Options and Content-Security-Policy are allowed by default; other headers must be listed in the global-allowed-response-headers option of spec.config, which is added to the default list.

Annotations together with snippets

An Ingress is rejected if one of the annotations below and a snippet annotation of the same Ingress set the same directive. NGINX treats most directives set twice in one location as a configuration error, and merges or ignores the second value of the others.

Comments in a snippet are ignored, but a directive name inside a quoted value is taken for a directive. The location-snippet option of spec.config is inserted into every location without such a check, so it must not set these directives either.

HSTS: nginx.ingress.kubernetes.io/ingress-nginx-hsts

Before (Critical):

nginx.ingress.kubernetes.io/configuration-snippet: |
  add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always;

After (not checked):

nginx.ingress.kubernetes.io/ingress-nginx-hsts: "true"

The annotation sets the Strict-Transport-Security: max-age=31536000; includeSubDomains header on every response of the locations of the Ingress. To change the policy, use the following annotations (Low), added in module version v1.3.0. An Ingress that sets any of them without ingress-nginx-hsts: "true" is rejected:

• nginx.ingress.kubernetes.io/ingress-nginx-hsts-max-age: seconds, from 0 to 63072000;
• nginx.ingress.kubernetes.io/ingress-nginx-hsts-include-subdomains: true by default;
• nginx.ingress.kubernetes.io/ingress-nginx-hsts-preload: false by default; true requires includeSubDomains and a max-age of at least 31536000.

nginx.ingress.kubernetes.io/ingress-nginx-hsts: "true"
nginx.ingress.kubernetes.io/ingress-nginx-hsts-max-age: "63072000"
nginx.ingress.kubernetes.io/ingress-nginx-hsts-preload: "true"

Browsers apply HSTS to the whole host, so use the same policy in all Ingresses of a host.

The annotation takes precedence over the controller-wide hsts option, which does not apply to the locations of the Ingress, and replaces a Strict-Transport-Security header sent by the backend or set by nginx.ingress.kubernetes.io/custom-headers. For particular response codes, nginx.ingress.kubernetes.io/response-headers-by-status can still set another value.

Controller client certificate for the backend: nginx.ingress.kubernetes.io/proxy-ssl-use-controller-certificate

Before (Critical):

nginx.ingress.kubernetes.io/configuration-snippet: |
  proxy_ssl_certificate /etc/nginx/ssl/client.crt;
  proxy_ssl_certificate_key /etc/nginx/ssl/client.key;
  proxy_ssl_protocols TLSv1.2;
  proxy_ssl_session_reuse on;

After (Low because of backend-protocol; the annotation itself is not checked):

nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
nginx.ingress.kubernetes.io/proxy-ssl-use-controller-certificate: "true"

The annotation adds these four directives to the locations of the Ingress, so that an HTTPS backend gets the client certificate the module issues for the controller. The certificate identifies the controller, so the annotation is accepted only on Deckhouse module Ingresses: the namespace starts with d8-, and the Ingress has the heritage: deckhouse label and a non-empty module label. The annotation cannot be combined with nginx.ingress.kubernetes.io/proxy-ssl-secret, which sets its own TLS parameters for the backend connection: such an Ingress is rejected.

Controller client certificate for external authentication: nginx.ingress.kubernetes.io/auth-proxy-ssl-use-controller-certificate

Added in module version v1.3.0.

Before (Critical):

nginx.ingress.kubernetes.io/auth-snippet: |
  proxy_ssl_certificate /etc/nginx/ssl/client.crt;
  proxy_ssl_certificate_key /etc/nginx/ssl/client.key;

After (High because of auth-url; the annotation itself is Medium):

nginx.ingress.kubernetes.io/auth-url: "https://auth.example.com/check"
nginx.ingress.kubernetes.io/auth-proxy-ssl-use-controller-certificate: "true"

The annotation adds proxy_ssl_certificate and proxy_ssl_certificate_key with the controller certificate to the internal location that sends the authentication request; the connection to the backend does not change. It requires an https nginx.ingress.kubernetes.io/auth-url and, like nginx.ingress.kubernetes.io/proxy-ssl-use-controller-certificate, is accepted only on Deckhouse module Ingresses.

TLS of external authentication: nginx.ingress.kubernetes.io/auth-proxy-ssl-protocols, nginx.ingress.kubernetes.io/auth-proxy-ssl-server-name, nginx.ingress.kubernetes.io/auth-proxy-ssl-name

Added in module version v1.3.0.

Before (Critical):

nginx.ingress.kubernetes.io/auth-snippet: |
  proxy_ssl_protocols TLSv1.2 TLSv1.3;
  proxy_ssl_server_name off;
  proxy_ssl_name auth.internal;

After (High because of auth-url; the annotations themselves are Low):

nginx.ingress.kubernetes.io/auth-url: "https://auth.example.com/check"
nginx.ingress.kubernetes.io/auth-proxy-ssl-protocols: "TLSv1.2 TLSv1.3"
nginx.ingress.kubernetes.io/auth-proxy-ssl-server-name: "false"
nginx.ingress.kubernetes.io/auth-proxy-ssl-name: "auth.internal"

The annotations set proxy_ssl_protocols, proxy_ssl_server_name (on by default) and proxy_ssl_name in the authentication location. The protocols are TLSv1, TLSv1.1, TLSv1.2 and TLSv1.3; the name must be a valid DNS name. The annotations require an https nginx.ingress.kubernetes.io/auth-url.

Timeouts of external authentication: nginx.ingress.kubernetes.io/auth-proxy-connect-timeout, nginx.ingress.kubernetes.io/auth-proxy-send-timeout, nginx.ingress.kubernetes.io/auth-proxy-read-timeout

Added in module version v1.3.0.

Before (Critical):

nginx.ingress.kubernetes.io/auth-snippet: |
  proxy_connect_timeout 2;
  proxy_send_timeout 2;
  proxy_read_timeout 2;

After (Low):

nginx.ingress.kubernetes.io/auth-proxy-connect-timeout: "2"
nginx.ingress.kubernetes.io/auth-proxy-send-timeout: "2"
nginx.ingress.kubernetes.io/auth-proxy-read-timeout: "2"

The annotations set proxy_connect_timeout, proxy_send_timeout and proxy_read_timeout of the authentication location, in seconds from 1 to 3600. By default the connection timeout is 15 seconds and the send and read timeouts are 60 seconds. The nginx.ingress.kubernetes.io/proxy-*-timeout annotations apply to the backend only. If the authentication request times out, the client gets 500.

Authentication response headers for the client: nginx.ingress.kubernetes.io/auth-response-headers-to-client

Added in module version v1.3.0.

Before (Critical):

nginx.ingress.kubernetes.io/configuration-snippet: |
  auth_request_set $token_rotation $upstream_http_x_token_rotation_pending;
  add_header X-Token-Rotation-Pending $token_rotation always;

After (Medium):

nginx.ingress.kubernetes.io/auth-response-headers-to-client: "X-Token-Rotation-Pending"

The annotation takes a comma-separated list of headers of the authentication response and adds them to the response to the client, whatever its status. A header the authentication service does not return is not added. To pass headers to the backend instead, use the standard nginx.ingress.kubernetes.io/auth-response-headers annotation. The headers are added with add_header: if nginx.ingress.kubernetes.io/custom-headers or nginx.ingress.kubernetes.io/response-headers-by-status sets a header with the same name, the client gets both values.

Authentication error codes: nginx.ingress.kubernetes.io/auth-error-status-codes

Added in module version v1.3.0.

Before (Critical):

nginx.ingress.kubernetes.io/configuration-snippet: |
  auth_request_set $auth_status $upstream_status;
  error_page 500 = @auth_error;
nginx.ingress.kubernetes.io/server-snippet: |
  location @auth_error {
    internal;
    if ($auth_status = 429) { return 429; }
    return 500;
  }

After (Low):

nginx.ingress.kubernetes.io/auth-error-status-codes: "429"

When the authentication service answers with a code other than 2xx, 401 or 403, NGINX returns 500 to the client. The annotation takes a comma-separated list of codes from 400 to 599, except 401 and 403, that are returned to the client as is. The controller generates the variable, the error_page and the internal location itself. These responses also get the headers of nginx.ingress.kubernetes.io/response-headers-by-status and nginx.ingress.kubernetes.io/ingress-nginx-hsts. Custom error pages keep working: if nginx.ingress.kubernetes.io/custom-http-errors or, without it, customErrors lists a code returned this way, including the 500 returned for other codes, the client gets the custom page instead, without these headers.

Request headers for the backend: nginx.ingress.kubernetes.io/proxy-request-headers

Added in module version v1.3.0.

Before (Critical):

nginx.ingress.kubernetes.io/configuration-snippet: |
  proxy_set_header X-Cluster-Id 00ce4b6f-5031-4483-836e-0a10db23fb21;
  proxy_set_header X-Request-Context /$1/$2;

After (Medium):

nginx.ingress.kubernetes.io/use-regex: "true"
nginx.ingress.kubernetes.io/proxy-request-headers: |
  X-Cluster-Id: 00ce4b6f-5031-4483-836e-0a10db23fb21
  X-Request-Context: /$1/$2

The annotation takes one Name: value per line; each line becomes proxy_set_header with the value in quotes. Values are printable ASCII without " and \. The only $ allowed is $1-$9, the regex captures of the location path; other NGINX variables cannot be used. Captures require use-regex: "true" or nginx.ingress.kubernetes.io/rewrite-target, and every path of the Ingress must have them. The captures are saved as soon as the request enters the location, so later regex matches, such as nginx.ingress.kubernetes.io/rewrite-target, do not change them. Headers the controller sets itself are rejected: Host, Connection, X-Forwarded-*, X-Original-Forwarded-Host, X-Real-IP, X-Request-ID, the ssl-client-* headers of nginx.ingress.kubernetes.io/auth-tls-secret, Accept-Encoding, which nginx.ingress.kubernetes.io/response-html-head-injection clears, and others. So are the headers listed in nginx.ingress.kubernetes.io/auth-response-headers of the same Ingress, as their values come from the authentication service. If the external authentication set by the global-auth-url option of spec.config passes a header with the same name, the header of the annotation is not sent. A header of additionalHeaders with the same name is replaced by the value of the annotation in the locations of the Ingress.

Response headers by status: nginx.ingress.kubernetes.io/response-headers-by-status

Added in module version v1.3.0.

Before (Critical):

nginx.ingress.kubernetes.io/configuration-snippet: |
  more_set_headers -s 429 "Retry-After: 60";

After (Medium):

nginx.ingress.kubernetes.io/response-headers-by-status: |
  429 Retry-After: 60
  503 Retry-After: 30

The annotation takes one <status> Name: value per line, with a status from 200 to 599; each line becomes more_set_headers -s <status> "Name: value";. Values follow the rules of nginx.ingress.kubernetes.io/proxy-request-headers but cannot contain $. For headers on every response, use nginx.ingress.kubernetes.io/custom-headers. The headers are set after those of nginx.ingress.kubernetes.io/custom-headers and nginx.ingress.kubernetes.io/ingress-nginx-hsts, so for the listed codes the values of this annotation take precedence.

Rate limit codes: nginx.ingress.kubernetes.io/limit-req-status-code, nginx.ingress.kubernetes.io/limit-conn-status-code

Added in module version v1.3.0.

Before (Critical):

nginx.ingress.kubernetes.io/configuration-snippet: |
  limit_req_status 429;
  limit_conn_status 429;

After (Low):

nginx.ingress.kubernetes.io/limit-rps: "10"
nginx.ingress.kubernetes.io/limit-req-status-code: "429"
nginx.ingress.kubernetes.io/limit-conn-status-code: "429"

The annotations set the status code returned to requests rejected by the rate limits of the Ingress: limit-req-status-code applies to requests over the limit-rps or limit-rpm limit, limit-conn-status-code to connections over the limit-connections limit. The code must be between 400 and 599. Without the annotations, the controller-wide limit-req-status-code and limit-conn-status-code options of spec.config apply, 503 by default. The annotations do not affect nginx.ingress.kubernetes.io/global-rate-limit (controller version 1.10 only), whose code is set by the global-rate-limit-status-code option of spec.config.

Fetch Metadata protection: nginx.ingress.kubernetes.io/fetch-metadata-protection

Added in module version v1.3.0.

Before (Critical):

nginx.ingress.kubernetes.io/configuration-snippet: |
  set $deny_no_cors "";
  if ($http_sec_fetch_mode = "no-cors") { set $deny_no_cors "N"; }
  if ($http_sec_fetch_site ~ ^(same-site|cross-site)$) { set $deny_no_cors "${deny_no_cors}S"; }
  if ($deny_no_cors = "NS") { return 403; }
  set $deny_cross_site "";
  if ($request_method !~ ^(GET|HEAD|OPTIONS)$) { set $deny_cross_site "M"; }
  if ($http_sec_fetch_site ~ ^(same-site|cross-site)$) { set $deny_cross_site "${deny_cross_site}S"; }
  if ($deny_cross_site = "MS") { return 403; }

After (Low):

nginx.ingress.kubernetes.io/fetch-metadata-protection: "true"

The annotation denies a request if its Sec-Fetch-Site is in the list of sites and either its Sec-Fetch-Mode is in the list of denied modes or its method is not in the list of allowed methods. The defaults reproduce the snippet: requests from another site (same-site or cross-site) that are no-cors or use a method other than GET, HEAD and OPTIONS get 403; requests from the same origin and user navigations (Sec-Fetch-Site: none) are not affected. Requests without Sec-Fetch-Site are never denied.

The policy is changed with the following annotations. Each takes a comma-separated list in which a value may appear only once. An Ingress that sets any of them without fetch-metadata-protection: "true" is rejected:

• nginx.ingress.kubernetes.io/fetch-metadata-sites: the Sec-Fetch-Site values the policy applies to: same-origin, same-site, cross-site, none. Default: same-site, cross-site.
• nginx.ingress.kubernetes.io/fetch-metadata-deny-modes: the denied Sec-Fetch-Mode values: cors, navigate, no-cors, same-origin, websocket. Default: no-cors; "" denies no mode.
• nginx.ingress.kubernetes.io/fetch-metadata-allowed-methods: the methods that are not denied: GET, HEAD, POST, PUT, DELETE, CONNECT, OPTIONS, TRACE, PATCH. Default: GET, HEAD, OPTIONS; "" denies every method, * disables the method check. * together with an empty fetch-metadata-deny-modes is rejected, as it would deny nothing.
• nginx.ingress.kubernetes.io/fetch-metadata-status-code: the response code, from 400 to 499. Default: 403.

For example, to deny cross-site navigations with any method with 429:

nginx.ingress.kubernetes.io/fetch-metadata-protection: "true"
nginx.ingress.kubernetes.io/fetch-metadata-sites: "cross-site"
nginx.ingress.kubernetes.io/fetch-metadata-deny-modes: "navigate"
nginx.ingress.kubernetes.io/fetch-metadata-allowed-methods: "*"
nginx.ingress.kubernetes.io/fetch-metadata-status-code: "429"

To deny every cross-site request:

nginx.ingress.kubernetes.io/fetch-metadata-protection: "true"
nginx.ingress.kubernetes.io/fetch-metadata-sites: "cross-site"
nginx.ingress.kubernetes.io/fetch-metadata-allowed-methods: ""

For each distinct policy the controller generates map blocks once, shared by all Ingresses that use it; the location only checks the result with if (...) { return <status>; }. Only values from the fixed lists above get into the NGINX configuration.

The check runs before the allowlist, the rate limits, authentication and the redirects of permanent-redirect, temporal-redirect and ssl-redirect, so a denied request gets the code of the policy even where these would answer otherwise. A CORS preflight request answered by nginx.ingress.kubernetes.io/enable-cors is answered before the check and is never denied.

Redirects keeping the request: nginx.ingress.kubernetes.io/redirect-preserve

Added in module version v1.3.0.

Before (Critical):

nginx.ingress.kubernetes.io/configuration-snippet: |
  if ($host = 'old.example.com') {
    rewrite ^ https://new.example.com$request_uri permanent;
  }

After, a separate Ingress for old.example.com (Medium because of permanent-redirect; the annotation itself is Low):

nginx.ingress.kubernetes.io/permanent-redirect: "https://new.example.com"
nginx.ingress.kubernetes.io/redirect-preserve: "request-uri"

The annotation appends the original request to the target of nginx.ingress.kubernetes.io/permanent-redirect or nginx.ingress.kubernetes.io/temporal-redirect:

• request-uri: the path and query (return 301 https://new.example.com$request_uri;); the target must have no path, not even /;
• query: the query only ($is_args$args); the target must have no query.

For example, a redirect of the exact path /login that keeps the query:

# path: /login, pathType: Exact
nginx.ingress.kubernetes.io/permanent-redirect: "https://new.example.com/login"
nginx.ingress.kubernetes.io/redirect-preserve: "query"

HTML in the page: nginx.ingress.kubernetes.io/response-html-head-injection

Added in module version v1.3.0.

Before (Critical):

nginx.ingress.kubernetes.io/configuration-snippet: |
  sub_filter "</head>" "<script src=\"https://counter.example.com/tag.js\" async></script></head>";
  sub_filter_once on;
  sub_filter_types text/html;
  proxy_set_header Accept-Encoding "";

After (High):

apiVersion: v1
kind: ConfigMap
metadata:
  name: metrics-injection
data:
  content.html: |
    <script src="https://counter.example.com/tag.js" async></script>
---
# Ingress annotations
nginx.ingress.kubernetes.io/response-html-head-injection: "metrics-injection"

The annotation takes the name of a ConfigMap in the namespace of the Ingress, or namespace/name of a ConfigMap in another namespace, and inserts its keys into text/html responses:

• head-start.html: right after the first <head>;
• head-end.html or content.html: before the first </head>; only one of them may be set;
• body-end.html: before the first </body>.

The keys can be combined, and other keys of the ConfigMap are ignored. For example, a counter in the page head and a chat widget at the end of the page:

apiVersion: v1
kind: ConfigMap
metadata:
  name: page-injection
data:
  head-end.html: |
    <script src="https://counter.example.com/tag.js" async></script>
  body-end.html: |
    <script src="https://chat.example.com/widget.js" async></script>

The tags are matched case-insensitively, and <head> is found only without attributes. If a page has no such tag, that part of the HTML is not inserted. The annotation is meant for applications whose HTML cannot be changed, for example to add an analytics counter.

The HTML never gets into the NGINX configuration: the controller writes it to files, a Lua module of the controller reads them, and sub_filter inserts the content through a variable. NGINX can only read these files, not change them. A changed ConfigMap is applied automatically; files no configuration uses are removed after an hour. The HTML of all keys together is limited to 64 KiB of UTF-8 text. The backend is asked for an uncompressed response, which the insertion requires. If the ConfigMap is missing, sets none of the keys, sets both head-end.html and content.html or exceeds the limit, the page is served unchanged. A ConfigMap of another namespace is used only if annotationPolicy.allowCrossNamespaceResources is enabled; otherwise the controller does not read it, and the page is served unchanged as well.

The annotation is High: the inserted HTML runs in the origin of the application with all its permissions. If the application sends Content-Security-Policy, the policy must allow the inserted script.