In Deckhouse Kubernetes Platform, the Circuit Breaker mechanism is implemented using Istio (the istio module) and provides the following capabilities:

  • Temporarily exclude an endpoint from load balancing if the error limit is exceeded.
  • Configure limits on the number of TCP connections and the number of requests to a single endpoint.
  • Detect stuck requests and terminate them with an error code (HTTP request timeout).

Example Circuit Breaker configuration

To detect problematic endpoints, use the outlierDetection settings in the DestinationRule custom resource. The Outlier Detection algorithm is described in more detail in the Envoy documentation.

Example:

apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: reviews-cb-policy
spec:
  host: reviews.prod.svc.cluster.local
  trafficPolicy:
    connectionPool:
      tcp:
        maxConnections: 100 # Maximum number of connections to the host, total across all endpoints.
      http:
        maxRequestsPerConnection: 10 # The connection will be recreated after every 10 requests.
    outlierDetection:
      consecutive5xxErrors: 7 # Allows up to 7 errors (including `5xx`, TCP timeouts, and HTTP timeouts)
      interval: 5m            # within 5 minutes,
      baseEjectionTime: 15m   # after which the endpoint will be removed from load balancing for 15 minutes.

You can also use the VirtualService resource to configure HTTP timeouts. These timeouts are also taken into account when calculating endpoint error statistics.

Example:

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: my-productpage-rule
  namespace: myns
spec:
  hosts:
  - productpage
  http:
  - timeout: 5s
    route:
    - destination:
        host: productpage