The module lifecycle stage: General Availability

Use the metallb module to expose LoadBalancer Services in bare-metal clusters without a cloud load balancer. The module supports L2 and BGP modes.

Example of metallb usage in L2 LoadBalancer mode

Enable the metallb module.

Prepare the application to publish:

d8 k create deploy nginx --image=nginx

Create a MetalLoadBalancerClass:

apiVersion: network.deckhouse.io/v1alpha1
kind: MetalLoadBalancerClass
metadata:
  name: ingress
spec:
  addressPool:
    - 192.168.2.100-192.168.2.150
  isDefault: false
  l2:
    # Network interfaces that announce the assigned IP addresses.
    interfaces:
      - eth1
  nodeSelector:
    # Selects nodes that run the L2 load balancer.
    node-role.kubernetes.io/loadbalancer: ""
  type: L2

In this example, spec.nodeSelector selects nodes with the node-role.kubernetes.io/loadbalancer label. Make sure the nodes intended to announce the Service IP addresses have this label.

If spec.l2.interfaces is omitted or empty, the assigned IP addresses are announced from all network interfaces of the selected nodes. Specify the interfaces intended for address announcements.

Create a Service with the annotation and loadBalancerClass set:

apiVersion: v1
kind: Service
metadata:
  name: nginx-deployment
  annotations:
    network.deckhouse.io/l2-load-balancer-external-ips-count: "3"
spec:
  type: LoadBalancer
  # Selects the MetalLoadBalancerClass named ingress.
  # Can be omitted if the cluster has a MetalLoadBalancerClass with isDefault: true
  # and BGP LoadBalancer mode is not used.
  loadBalancerClass: ingress
  ports:
    - port: 8000
      protocol: TCP
      targetPort: 80
  selector:
    app: nginx

As a result, the Service of type LoadBalancer receives the specified number of IP addresses:

d8 k get svc

Output example:

NAME                   TYPE           CLUSTER-IP      EXTERNAL-IP                                 PORT(S)          AGE
nginx-deployment       LoadBalancer   10.222.130.11   192.168.2.100,192.168.2.101,192.168.2.102   8000:30544/TCP   11s

When creating a Service, you can also specify certain IP addresses from the pool that will be assigned to it.

You can use the assigned EXTERNAL-IP addresses in DNS A records for the application domain. Check that the application responds at these addresses:

$ curl -s -o /dev/null -w "%{http_code}" 192.168.2.100:8000
200
$ curl -s -o /dev/null -w "%{http_code}" 192.168.2.101:8000
200
$ curl -s -o /dev/null -w "%{http_code}" 192.168.2.102:8000
200

Running speakers on nodes with custom taints

In L2 LoadBalancer mode, the speaker runs as a DaemonSet on the nodes selected by the nodeSelector parameter of MetalLoadBalancerClass. The speaker tolerates only the following taints known to Deckhouse Platform (DP):

  • node-role.kubernetes.io/master
  • node-role.kubernetes.io/control-plane
  • node.deckhouse.io/etcd-arbiter
  • dedicated.deckhouse.io and dedicated with any value
  • DeletionCandidateOfClusterAutoscaler and ToBeDeletedByClusterAutoscaler

If a selected node has a taint with any other key, the speaker is not scheduled on it, and the addresses are not announced from that node.

To run the speaker on such nodes, add the taint key to the customTolerationKeys parameter of the global ModuleConfig. The key must match the taint key set on the nodes, for example, in the spec.nodeTemplate.taints parameter of NodeGroup. Adding a key there allows both the speaker and the other system components (CNI, CSI, monitoring) to run on such nodes:

apiVersion: deckhouse.io/v1alpha1
kind: ModuleConfig
metadata:
  name: global
spec:
  version: 3
  settings:
    modules:
      placement:
        customTolerationKeys:
          - dedicated.example.com

Check that the speaker pods are running on the tainted nodes:

d8 k -n d8-metallb get pods -l app=l2lb-speaker -o wide

The speaker in BGP LoadBalancer mode has the same tolerations, so this setting applies to it as well.

Example of metallb usage in BGP LoadBalancer mode

Enable the metallb module and create the following objects for BGP mode:

apiVersion: network.deckhouse.io/v1alpha1
kind: MetalLoadBalancerPool
metadata:
  name: mypool
spec:
  addresses:
    - 192.168.219.100-192.168.219.200
---
apiVersion: network.deckhouse.io/v1alpha1
kind: MetalLoadBalancerBGPPeer
metadata:
  name: myrouter
spec:
  peerAddress: 172.18.18.10
  peerASN: 64601
  myASN: 64600
  holdTime: 3s
---
apiVersion: network.deckhouse.io/v1alpha1
kind: MetalLoadBalancerConfiguration
metadata:
  name: defaults
spec:
  mode: BGP
  nodeSelector:
    node-role.deckhouse.io/metallb: ""
  bgp:
    peerNames:
      - myrouter
  advertisements:
    - poolNames:
        - mypool

In this example, spec.nodeSelector selects nodes with the node-role.deckhouse.io/metallb label. Make sure the nodes intended to run the speaker pods have this label.

Configure BGP peering on the network equipment.

Example of using L2 LoadBalancer and BGP LoadBalancer modes simultaneously

The module can serve services in L2 LoadBalancer mode and BGP LoadBalancer mode at the same time. The modes are served by different sets of components: l2lb-controller and l2lb-speaker in L2 LoadBalancer mode, controller and speaker in BGP LoadBalancer mode. The sets run independently, but they process the same services and see each other’s address pools, so the modes can work together only in the configuration described below.

BGP LoadBalancer mode is available in the Enterprise, Ultimate, or Certified Security Pro editions.

The configuration described below is the only supported way to run L2 LoadBalancer and BGP LoadBalancer modes simultaneously. Any deviation from it (an L2 service without spec.loadBalancerClass, a MetalLoadBalancerClass with the isDefault: true parameter, overlapping address pools, manually created MetalLB objects) makes the components of the modes conflict with each other, which leads to service unavailability or exhaustion of the L2 LoadBalancer pool.

Enable the module and configure both modes:

apiVersion: network.deckhouse.io/v1alpha1
kind: MetalLoadBalancerClass
metadata:
  name: ingress
spec:
  addressPool:
    - 192.168.2.100-192.168.2.150
  # A default class is not allowed.
  isDefault: false
  l2:
    # Network interfaces that announce the assigned IP addresses.
    interfaces:
      - eth1
  nodeSelector:
    node-role.kubernetes.io/loadbalancer: ""
  type: L2
---
apiVersion: network.deckhouse.io/v1alpha1
kind: MetalLoadBalancerPool
metadata:
  name: mypool
spec:
  # The addresses do not overlap with the MetalLoadBalancerClass pool.
  addresses:
    - 192.168.219.100-192.168.219.200
---
apiVersion: network.deckhouse.io/v1alpha1
kind: MetalLoadBalancerBGPPeer
metadata:
  name: myrouter
spec:
  peerAddress: 172.18.18.10
  peerASN: 64601
  myASN: 64600
  holdTime: 3s
---
apiVersion: network.deckhouse.io/v1alpha1
kind: MetalLoadBalancerConfiguration
metadata:
  name: defaults
spec:
  mode: BGP
  nodeSelector:
    node-role.deckhouse.io/metallb: ""
  bgp:
    peerNames:
      - myrouter
  advertisements:
    - poolNames:
        - mypool

Create an L2 LoadBalancer service with MetalLoadBalancerClass specified explicitly:

apiVersion: v1
kind: Service
metadata:
  name: nginx-l2
  annotations:
    network.deckhouse.io/l2-load-balancer-external-ips-count: "2"
spec:
  type: LoadBalancer
  # MetalLoadBalancerClass name.
  loadBalancerClass: ingress
  ports:
    - port: 8000
      protocol: TCP
      targetPort: 80
  selector:
    app: nginx

Create a BGP LoadBalancer service without spec.loadBalancerClass and with the pool specified explicitly:

apiVersion: v1
kind: Service
metadata:
  name: nginx-bgp
  annotations:
    # MetalLoadBalancerPool name.
    network.deckhouse.io/load-balancer-pool: mypool
spec:
  type: LoadBalancer
  ports:
    - port: 8000
      protocol: TCP
      targetPort: 80
  selector:
    app: nginx

As a result, the nginx-l2 service will be assigned two addresses from the MetalLoadBalancerClass pool, and the nginx-bgp service will be assigned one address from MetalLoadBalancerPool:

d8 k get svc

Output example:

NAME        TYPE           CLUSTER-IP      EXTERNAL-IP                   PORT(S)          AGE
nginx-bgp   LoadBalancer   10.222.41.120   192.168.219.100               8000:31942/TCP   12s
nginx-l2    LoadBalancer   10.222.130.11   192.168.2.100,192.168.2.101   8000:30544/TCP   15s

Limitations when running the modes simultaneously

When both modes run in the cluster, the following limitations apply:

  • Each L2 LoadBalancer service must specify MetalLoadBalancerClass explicitly in spec.loadBalancerClass. The BGP LoadBalancer controller processes services without this parameter, and a service with multiple addresses stops receiving traffic until it is recreated (although the nodes keep responding to ARP requests for its addresses).
  • No MetalLoadBalancerClass may have the isDefault: true parameter set. Otherwise, the L2 LoadBalancer controller allocates a hidden address from its pool for each BGP LoadBalancer service, and the L2 LoadBalancer pool is silently exhausted.
  • A BGP LoadBalancer service must not specify spec.loadBalancerClass and is assigned only one IPv4 address. Services with multiple addresses are supported only in L2 LoadBalancer mode.
  • Specify a pool for a BGP LoadBalancer service with the network.deckhouse.io/load-balancer-pool annotation or an address with the network.deckhouse.io/load-balancer-ips annotation. The BGP LoadBalancer controller also sees the MetalLoadBalancerClass pools, so without an annotation it may assign the service an address from the L2 LoadBalancer pool that is not announced via BGP.
  • Addresses in MetalLoadBalancerPool must not overlap with addresses in the addressPool parameter of MetalLoadBalancerClass. If they overlap, the speaker components do not apply the configuration and report the CIDR ... overlaps with already defined CIDR error.
  • A service can be moved from L2 LoadBalancer mode to BGP LoadBalancer mode only by following the migration procedure.
  • The controller logs may contain messages like PoolStatusController ... the object has been modified. They do not affect traffic.

Example of migration from L2 LoadBalancer mode to BGP LoadBalancer mode

During the migration, the L2 LoadBalancer components and the BGP LoadBalancer components run in the cluster simultaneously, and the limitations of simultaneous operation apply. Both controllers can process the same services:

  • The BGP LoadBalancer controller processes all services of the LoadBalancer type that have no spec.loadBalancerClass parameter set.
  • The L2 LoadBalancer controller processes services whose spec.loadBalancerClass is the name of a MetalLoadBalancerClass. If the cluster has a MetalLoadBalancerClass with the isDefault: true parameter, the L2 LoadBalancer controller also processes all other services of the LoadBalancer type.

The examples below use the l2-default default class. The module creates a class with this name when it converts legacy module settings. Your cluster may have a default class with a different name. Find it and use its name instead of l2-default:

d8 k get metalloadbalancerclass -o custom-columns='NAME:.metadata.name,DEFAULT:.spec.isDefault'

The procedure described below is the only supported way to migrate from L2 LoadBalancer mode to BGP LoadBalancer mode. Enabling BGP LoadBalancer mode without preparing L2 services, having an address in the pools of both modes at the same time, and manually creating MetalLB objects (IPAddressPool, BGPAdvertisement, etc.) lead to service unavailability.

The migration involves a brief unavailability: less than 1 second when recreating a service in step 1 and about 11 seconds for each service moved to BGP LoadBalancer mode in step 4.

Before you begin

Make sure that the cluster and the network meet the following requirements:

  • The cluster runs the Enterprise, Ultimate, or Certified Security Pro edition. BGP LoadBalancer mode is not available in other editions.
  • The network equipment accepts /32 BGP announcements from the nodes for addresses from the L2 LoadBalancer subnet. Such a route has a longer prefix, so the router prefers it over the route to the directly connected subnet.
  • Clients of the moved services are outside the L2 LoadBalancer subnet, or the network sends their traffic to these addresses through the router. Clients in the same subnet resolve the service address via ARP and do not use the router. After the move, the nodes no longer respond to ARP requests for the address, and such clients lose access to the service.

Step 1. Binding L2 services to MetalLoadBalancerClass

If you enable BGP LoadBalancer mode while L2 services have no spec.loadBalancerClass set, the BGP LoadBalancer controller starts processing them. Services with multiple addresses (the network.deckhouse.io/l2-load-balancer-external-ips-count annotation is greater than 1) stop receiving traffic: the controller cannot assign multiple IPv4 addresses to a service and clears its status, while the nodes keep responding to ARP requests for these addresses. Disabling BGP LoadBalancer mode does not restore such services, only recreating them helps.

Find services of the LoadBalancer type that have no spec.loadBalancerClass set (<none> in the CLASS column):

d8 k get svc -A -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name,TYPE:.spec.type,CLASS:.spec.loadBalancerClass,EXTERNAL-IP:.status.loadBalancer.ingress[*].ip' | awk 'NR==1 || $3=="LoadBalancer"'

Output example:

NAMESPACE   NAME               TYPE           CLASS    EXTERNAL-IP
default     nginx-deployment   LoadBalancer   <none>   192.168.2.100,192.168.2.101
default     web                LoadBalancer   <none>   192.168.2.120

Recreate each service that should remain in L2 LoadBalancer mode with spec.loadBalancerClass set explicitly. This parameter cannot be changed for an existing service. To keep the current addresses of the service, list them in the network.deckhouse.io/load-balancer-ips annotation.

Save the service manifest to the nginx-deployment.yaml file:

apiVersion: v1
kind: Service
metadata:
  name: nginx-deployment
  namespace: default
  annotations:
    network.deckhouse.io/l2-load-balancer-external-ips-count: "2"
    network.deckhouse.io/load-balancer-ips: "192.168.2.100,192.168.2.101"
spec:
  type: LoadBalancer
  # MetalLoadBalancerClass name.
  loadBalancerClass: l2-default
  ports:
    - port: 8000
      protocol: TCP
      targetPort: 80
  selector:
    app: nginx

Delete the service and create it again from the saved manifest:

d8 k -n default delete svc nginx-deployment && d8 k apply -f nginx-deployment.yaml

Do not set spec.loadBalancerClass for services that will be moved to BGP LoadBalancer mode. It is enough to pin their current address with the network.deckhouse.io/load-balancer-ips annotation, no recreation is required:

d8 k -n default annotate svc web network.deckhouse.io/load-balancer-ips=192.168.2.120

In BGP LoadBalancer mode, a service is assigned only one IPv4 address, so only services with a single address can be moved to this mode. Keep services with multiple addresses in L2 LoadBalancer mode.

Step 2. Disabling the default MetalLoadBalancerClass

While MetalLoadBalancerClass has the isDefault: true parameter set, the L2 LoadBalancer controller also processes BGP LoadBalancer services. For each such service, it allocates an address from the MetalLoadBalancerClass pool and announces it via ARP, although this address is not shown in the service status. Such hidden addresses silently consume the L2 LoadBalancer pool and can lead to its exhaustion. Services with an address pinned by the network.deckhouse.io/load-balancer-ips annotation are not affected by this issue.

Set the isDefault parameter to false:

d8 k patch metalloadbalancerclass l2-default --type=merge -p '{"spec":{"isDefault":false}}'

The parameter affects only new services. Services that have already been bound to the default MetalLoadBalancerClass remain bound to it.

Addresses allocated in L2 LoadBalancer mode are represented by SDNInternalL2LBService objects named <service name>-<MetalLoadBalancerClass name>-<index>:

d8 k get sdninternall2lbservices -A

If BGP LoadBalancer mode has already been enabled earlier and the list contains objects for services of this mode, these services received a hidden address from the L2 LoadBalancer pool before isDefault was disabled. Disabling isDefault does not release such an address, since the service remains bound to MetalLoadBalancerClass. Specify the current address of such a service from the BGP LoadBalancer pool in its network.deckhouse.io/load-balancer-ips annotation. This address is not in the L2 LoadBalancer pool, so the L2 LoadBalancer controller releases the hidden address, and the service address does not change. An example for the web-bgp service with the 192.168.219.100 address:

d8 k -n default annotate svc web-bgp network.deckhouse.io/load-balancer-ips=192.168.219.100

Step 3. Enabling BGP LoadBalancer mode

Create a MetalLoadBalancerPool, a MetalLoadBalancerBGPPeer, and a MetalLoadBalancerConfiguration as shown in the BGP LoadBalancer example. Addresses in MetalLoadBalancerPool must not overlap with addresses in the addressPool parameter of MetalLoadBalancerClass.

Make sure that the components of both modes are running in the module:

d8 k -n d8-metallb get pods

Check that the addresses of L2 services have not changed:

d8 k get svc -A

While both modes are running simultaneously, the controller logs may contain messages like PoolStatusController ... "l2-default": the object has been modified. They do not affect traffic.

Step 4. Moving a single-address service to BGP LoadBalancer mode

Move services one at a time. With a single d8 k apply command, remove the service address from the addressPool parameter of MetalLoadBalancerClass and add it to the addresses parameter of MetalLoadBalancerPool. The address must not be in the pools of both modes at the same time: in this case, the speaker components cannot apply the configuration due to overlapping pools (CIDR "192.168.2.120/32" in pool "l2-default" overlaps with already defined CIDR).

Prepare a file with the MetalLoadBalancerClass and MetalLoadBalancerPool manifests. Take the current parameters of both objects from the cluster and change only the address lists:

d8 k get metalloadbalancerclass l2-default -o yaml
d8 k get metalloadbalancerpool mypool -o yaml

Put MetalLoadBalancerClass first in the file. d8 k apply applies the manifests in the order they appear, so the address is removed from the L2 LoadBalancer pool before it is added to MetalLoadBalancerPool.

An example of moving the 192.168.2.120 address that belongs to the 192.168.2.100-192.168.2.150 range. The migrate-192.168.2.120.yaml file:

apiVersion: network.deckhouse.io/v1alpha1
kind: MetalLoadBalancerClass
metadata:
  name: l2-default
spec:
  addressPool:
    - 192.168.2.100-192.168.2.119
    - 192.168.2.121-192.168.2.150
  isDefault: false
  # Keep the current values of the parameters.
  l2:
    interfaces:
      - eth1
  nodeSelector:
    node-role.kubernetes.io/loadbalancer: ""
  type: L2
---
apiVersion: network.deckhouse.io/v1alpha1
kind: MetalLoadBalancerPool
metadata:
  name: mypool
spec:
  addresses:
    - 192.168.219.100-192.168.219.200
    - 192.168.2.120/32

Apply both manifests with a single command:

d8 k apply -f migrate-192.168.2.120.yaml

The L2 LoadBalancer controller releases the address, after which the BGP LoadBalancer controller assigns it to the service and starts announcing it via BGP. Check the service events:

d8 k -n default describe svc web

The service events should show, in order, ClearAssignment (the address is removed from the L2 LoadBalancer pool), IPAllocated (the address is assigned by the BGP LoadBalancer controller), and nodeAssigned with protocol "bgp" (the address is announced via BGP).

Make sure that the router has a /32 route to the service address via the cluster nodes, and that the nodes no longer respond to ARP requests for this address.

Additional configuration examples for Service

After setting up a load balancer, use Service annotations to control address allocation and sharing.

Creating a Service and assigning it specific IP addresses from the pool

To assign specific addresses to a Service, list them in the network.deckhouse.io/load-balancer-ips annotation. To assign more than one address, also set the network.deckhouse.io/l2-load-balancer-external-ips-count annotation to the number of addresses to allocate from the pool. The value must not be less than the number of addresses in network.deckhouse.io/load-balancer-ips; otherwise, the extra addresses are not assigned.

apiVersion: v1
kind: Service
metadata:
  name: nginx-deployment
  annotations:
    # The number of addresses that will be allocated from the pool declared in MetalLoadBalancerClass.
    network.deckhouse.io/l2-load-balancer-external-ips-count: "3"
    # A list of addresses from the pool declared in MetalLoadBalancerClass that will be allocated to the Service.
    network.deckhouse.io/load-balancer-ips: "192.168.2.102,192.168.2.103,192.168.2.104"
spec:
  type: LoadBalancer
  # Selects the MetalLoadBalancerClass named ingress.
  # Can be omitted if the cluster has a MetalLoadBalancerClass with isDefault: true
  # and BGP LoadBalancer mode is not used.
  loadBalancerClass: ingress
  ports:
    - port: 8000
      protocol: TCP
      targetPort: 80
  selector:
    app: nginx

As a result, the Service of type LoadBalancer receives the specified IP addresses:

d8 k get svc

Output example:

NAME                   TYPE           CLUSTER-IP      EXTERNAL-IP                                 PORT(S)          AGE
nginx-deployment       LoadBalancer   10.222.130.11   192.168.2.102,192.168.2.103,192.168.2.104   8000:30544/TCP   11s

Creating a Service with a specific IP address

To assign a specific IP address to a Service, add the network.deckhouse.io/load-balancer-ips annotation:

apiVersion: v1
kind: Service
metadata:
  name: nginx
  annotations:
    network.deckhouse.io/load-balancer-ips: 192.168.2.105
spec:
  loadBalancerClass: ingress
  ports:
    - port: 80
      targetPort: 80
  selector:
    app: nginx
  type: LoadBalancer

Creating Services with shared IP addresses

To allow multiple Services to share an IP address, add the network.deckhouse.io/load-balancer-shared-ip-key annotation to each Service. Its value is the sharing key. Services can share an IP address under the following conditions:

  • They have the same sharing key.
  • Their protocol and port combinations do not conflict (for example, TCP/53 and UDP/53 can use the same IP address).
  • They use externalTrafficPolicy: Cluster, or they both point to the exact same set of pods (that is, the pod selectors are identical).

Even when these conditions are met, MetalLB does not guarantee that the Services will receive the same IP address. To request a specific shared address, set network.deckhouse.io/load-balancer-ips on both Services.

The following example configures two Services to share an IP address:

apiVersion: v1
kind: Service
metadata:
  name: dns-service-tcp
  namespace: default
  annotations:
    network.deckhouse.io/load-balancer-shared-ip-key: "key-to-share-1.2.3.4"
spec:
  type: LoadBalancer
  loadBalancerClass: ingress
  ports:
    - name: dnstcp
      protocol: TCP
      port: 53
      targetPort: 53
  selector:
    app: dns
---
apiVersion: v1
kind: Service
metadata:
  name: dns-service-udp
  namespace: default
  annotations:
    network.deckhouse.io/load-balancer-shared-ip-key: "key-to-share-1.2.3.4"
spec:
  type: LoadBalancer
  loadBalancerClass: ingress
  ports:
    - name: dnsudp
      protocol: UDP
      port: 53
      targetPort: 53
  selector:
    app: dns

Assigning an IP address pool to a Service in BGP LoadBalancer mode

In BGP LoadBalancer mode, assign an IP address pool to a Service using the network.deckhouse.io/load-balancer-pool annotation. For L2 LoadBalancer mode, use the MetalLoadBalancerClass settings, as shown in the L2 LoadBalancer mode example.

apiVersion: v1
kind: Service
metadata:
  name: nginx
  annotations:
    network.deckhouse.io/load-balancer-pool: mypool
spec:
  ports:
    - port: 80
      targetPort: 80
  selector:
    app: nginx
  type: LoadBalancer