Available in editions: Open/CE, BE, SE, SE+, Ultimate/EE, Core
The module lifecycle stage: General Availability
Every 15 minutes, the module analyzes the cluster state and performs pod eviction according to the conditions described in the active strategies. Evicted pods go through the scheduling process again, considering the current state of the cluster. This helps redistribute workloads according to the chosen strategy.
The module is based on the descheduler project.
Features of the module
- The module can take into account the pod priority class (parameter spec.priorityClassThreshold), restricting its operation to only those pods that have a priority class lower than the specified threshold;
- The module does not evict pods in the following cases:
- a pod is in the
d8-*orkube-systemnamespaces; - a pod has a
priorityClassNamesystem-cluster-criticalorsystem-node-critical; - a pod is associated with a local storage;
- a pod is associated with a DaemonSet;
- pod eviction will violate Pod Disruption Budget (PDB);
- there are no available nodes to run the evicted pod.
- a pod is in the
- Pods with the
Best effortpriority class are evicted before those withBurstableandGuaranteed.
Descheduler uses parameters with the labelSelector syntax from Kubernetes to filter pods and nodes:
podLabelSelector— limits pods by labels;namespaceLabelSelector— filters pods by namespaces;nodeLabelSelector— selects nodes by labels.
Metrics provider
If the cluster has a registered metrics.k8s.io API group (for example, when metrics-server is installed), the module automatically detects it and enables the KubernetesMetrics metrics provider in the descheduler policy. This allows to use the actual resource consumption data from the Metrics API instead of relying solely on pod requests and limits when the LowNodeUtilization strategy is set. Accordingly, the values in thresholds and targetThresholds are then interpreted as a percentage of actual consumption, not of pod requests. The HighNodeUtilization strategy is not affected: it always uses pod requests and limits.
No user action is required: the module discovers the metrics.k8s.io API group by watching APIService resources and configures the policy accordingly. If metrics-server is installed after the module is already running, the descheduler Pod will be automatically restarted with the updated policy.
If the metrics.k8s.io API group is not registered in the cluster, the module estimates resource usage from pod requests and limits.
Evictions in background
The module runs the descheduler with the upstream EvictionsInBackground feature gate enabled (--feature-gates EvictionsInBackground=true). There is no module setting for it: the behavior is part of the module and requires no configuration.
By default, the descheduler expects every eviction to complete immediately: the eviction API request either succeeds or is rejected. Some workloads cannot be evicted that way — for example, a pod backing a virtual machine has to be live-migrated first, which may take minutes and may need retries.
With the feature gate enabled, an eviction of a pod annotated with descheduler.alpha.kubernetes.io/request-evict-only is treated as a request to start the eviction. The controller that owns the pod rejects the eviction API call with 429 Too Many Requests and marks the pod with the descheduler.alpha.kubernetes.io/eviction-in-progress annotation while it performs the actual eviction (for example, a live migration). The descheduler keeps track of such pods and counts them against its eviction limits until the pod is gone or the annotation is removed, so it doesn’t start more evictions than allowed while the previous ones are still in progress.
Pods without the descheduler.alpha.kubernetes.io/request-evict-only annotation are evicted as usual, so nothing changes for clusters where no workload implements its own eviction policy.
The feature gate is in the alpha stage in the upstream descheduler project (see KEP-1397). It changes nothing on its own: without a controller that answers the eviction request as described above, the descheduler behaves exactly as it did before.
Strategies
HighNodeUtilization
More compactly places pods. Requires scheduler configuration and enabling auto-scaling.
To use HighNodeUtilization, you must explicitly specify the high-node-utilization scheduler profile for each pod (this profile cannot be set as the default).
This strategy identifies under utilized nodes and evicts pods from them to redistribute them more compactly across fewer nodes.
Under utilized node — A node whose resource usage is below all the threshold values specified in the strategies.highNodeUtilization.thresholds section.
The strategy is enabled by the parameter spec.strategies.highNodeUtilization.enabled.
In GKE, you cannot configure the default scheduler, but you can use the optimize-utilization strategy or deploy a second custom scheduler.
Node resource usage takes into account extended resources and is calculated based on pod requests and limits (requests and limits), not actual consumption. This approach ensures consistency with the kube-scheduler, which uses a similar principle when scheduling pods on nodes.
This strategy always uses pod requests and limits, regardless of whether the metrics.k8s.io API is available in the cluster (see Metrics provider).
LowNodeUtilization
Loads the nodes more evenly.
This strategy identifies under utilized nodes and evicts pods from other over utilized nodes. The strategy assumes that the evicted pods will be recreated on the under utilized nodes (following normal scheduler behavior).
Under utilized node — A node whose resource usage is below all the threshold values specified in the strategies.lowNodeUtilization.thresholds section.
Over utilized node — A node whose resource usage exceeds at least one of the threshold values specified in the strategies.lowNodeUtilization.targetThresholds section.
Nodes with resource usage in the range between thresholds and targetThresholds are considered optimally utilized. Pods on these nodes will not be evicted.
The strategy is enabled by the parameter spec.strategies.lowNodeUtilization.enabled.
By default, thresholds and targetThresholds are absolute percentages. If spec.strategies.lowNodeUtilization.useDeviationThresholds is true, they are counted from the average utilization of all Ready nodes of the cluster: a node is under utilized if its usage is at or below the average minus thresholds, and over utilized if its usage is above the average plus targetThresholds. For example, with an average CPU utilization of 40%, thresholds: {cpu: 5} and targetThresholds: {cpu: 5} make nodes at or below 35% under utilized and nodes above 45% over utilized, so the strategy keeps the nodes within the corridor of ±5% around the average.
To evict pods gradually, set spec.strategies.lowNodeUtilization.evictionLimits.node: the maximum number of pods the strategy evicts from one node in one descheduling cycle. A virtual machine that is still migrating from the node takes a place in the limit in every cycle, so for virtual machines the parameter effectively limits how many of them migrate from one node at the same time.
If the metrics.k8s.io API group is not registered in the cluster, node resource usage takes into account extended resources and is calculated based on pod requests and limits (requests and limits), not actual consumption. This approach ensures consistency with the kube-scheduler, which uses a similar principle when scheduling pods on nodes.
If the metrics.k8s.io API group is registered (for details, see “Metrics provider”), the strategy uses the actual resource consumption from the Metrics API instead of requests and limits. In this case the values used by the descheduler are close to what kubectl top shows, but they no longer match the kube-scheduler’s view: a node whose requests are exhausted may still be considered under utilized, and its pods will not be evicted.
CPU utilization averaged over 15 minutes
The momentary values from the Metrics API or pod requests react to short load spikes. To balance nodes by their sustained load, set spec.strategies.lowNodeUtilization.metricsSource to Prometheus. The strategy then uses the node CPU utilization averaged over the last 15 minutes, which the cluster Prometheus calculates from the node-exporter data.
How the strategy works in this mode:
- Only the
cpukey ofthresholdsandtargetThresholdsis used, as a percentage of the node CPU. The key is required in both sections, other keys are ignored. - The
prometheusmodule must be enabled. If it is disabled, the strategy works as withDefault, and the Deckhouse log shows a warning. - With
useDeviationThresholds: truethe strategy keeps the nodes within a corridor around the 15-minute average CPU utilization of the cluster. - The strategy cannot estimate how much one pod adds to the average, so it evicts one pod from each over utilized node in a cycle. With
evictionLimits.nodeset, it evicts up to that number of pods from each over utilized node: in this mode the parameter raises the number of evictions instead of limiting it. Do not setevictionLimits.nodewith this source. - The average reflects the evictions with a delay of up to 15 minutes. With the
Frequentdescheduling interval a node stays over utilized for several cycles after the evictions and loses pods again. Use theModerate(15 minutes, default) orRare(30 minutes) interval with this source. - If Prometheus is unavailable or has no data for at least one Ready node (node-exporter does not run on it or fails to be scraped, or the node has just joined and has not been scraped twice yet), the strategy of this Descheduler skips the cycle. Other Descheduler resources keep working.
The descheduler reads the metrics through aggregating-proxy in the d8-monitoring namespace with its ServiceAccount token. The module grants the read access only while at least one Descheduler uses this mode. An example is on the “Examples” page.
Eviction target check
Since Deckhouse 1.78, in clusters with the control-plane-manager module enabled, the strategy requests the expected target node from kube-scheduler before evicting a pod. kube-scheduler runs a dry-run scheduling cycle for the pod, with the source node excluded, and returns the nodes with the top score.
How the strategy uses the answer:
- The pod is evicted only if every node with the top score is under utilized in this cycle by this Descheduler’s own
thresholds. WithuseDeviationThresholds: truethis means at or below the average minusthresholds. The check has no thresholds of its own. - The pod stays on the node, and the strategy tries the next pod of the same node, if kube-scheduler would place the pod on a node that is not under utilized or has no node for it.
- The pod also stays if kube-scheduler does not answer or answers with an error: the strategy does not evict a pod without the check.
- kube-scheduler pauses the real scheduling while a check runs, so the checks are limited. After 5 checks of one over utilized node that did not allow an eviction, the node waits for the next cycle. After 100 checks in a cycle, this Descheduler resource stops evicting pods until the next cycle. If the check finds no feasible node or any top-scoring node is not underutilized, other pods of the same controller on the source node are skipped for this cycle.
The check applies to every Descheduler with lowNodeUtilization and cannot be turned off. On managed Kubernetes (the control-plane-manager module is disabled) the strategy works without it. The descheduler_simulate_target_check_total metric counts the checks by result.
RemoveDuplicates
Prevents multiple pods from the same controller (ReplicaSet, ReplicationController, StatefulSet) or the same Job from running on the same node.
The strategy ensures that no more than one pod of a ReplicaSet, ReplicationController, StatefulSet, or pods of a single Job is running on the same node. If there are two or more such pods, the module evicts the excess pods so that they are better distributed across the cluster.
The situation can occur if some nodes in the cluster have failed for any reason, and the pods from those nodes have been moved to other nodes. Once the failed nodes become available again to accept load, this strategy can be used to evict duplicate pods from other nodes.
The strategy is enabled by the parameter strategies.removeDuplicates.enabled.
RemovePodsHavingTooManyRestarts
The strategy evicts pods having too many restarts from nodes.
Pods become eviction candidates when the total number of restarts across all containers, including init containers, exceeds the podRestartThreshold threshold.
This strategy is useful for evicting pods in the CrashLoopBackOff state or with repeated failures, as well as for freeing up resources and allowing fresh pods to be scheduled on potentially healthier nodes.
RemovePodsViolatingInterPodAntiAffinity
Evicts pods violating inter-pod affinity and anti-affinity rules to ensure compliance.
The strategy ensures that pods violating inter-pod affinity and anti-affinity rules are evicted from nodes.
For example, if there is podA on a node and podB and podC (running on the same node) have anti-affinity rules which prohibit them to run on the same node, then podA will be evicted from the node so that podB and podC could run. This issue could happen, when the anti-affinity rules for podB and podC are created when they are already running on node.
The strategy is enabled by the parameter spec.strategies.highNodeUtilization.enabled.
RemovePodsViolatingNodeAffinity
Evicts pods violating node affinity rules to ensure compliance.
The strategy makes sure all pods violating node affinity are eventually removed from nodes.
Essentially, depending on the settings of the parameter strategies.removePodsViolatingNodeAffinity.nodeAffinityType, the strategy temporarily implement the rule requiredDuringSchedulingIgnoredDuringExecution of the pod’s node affinity as the rule requiredDuringSchedulingRequiredDuringExecution, and the rule preferredDuringSchedulingIgnoredDuringExecution as the rule preferredDuringSchedulingPreferredDuringExecution.
Example for nodeAffinityType: requiredDuringSchedulingIgnoredDuringExecution. There is a pod scheduled to a node which satisfies the node affinity rule requiredDuringSchedulingIgnoredDuringExecution at the time of scheduling. If over time this node no longer satisfies the node affinity rule, and there is another node available that satisfies the node affinity rule, the strategy evicts the pod from the node it was originally scheduled to.
Example for nodeAffinityType: preferredDuringSchedulingIgnoredDuringExecution. There is a pod scheduled to a node because at the time of scheduling there were no other nodes that satisfied the node affinity rule preferredDuringSchedulingIgnoredDuringExecution. If over time an available node that satisfies this rule appears in the cluster, the strategy evicts the pod from the node it was originally scheduled to.
The strategy is enabled by the parameter strategies.removePodsViolatingNodeAffinity.enabled.
RemovePodsViolatingTopologySpreadConstraint
The strategy ensures that pods violating topology spread constraints are evicted from nodes. It evicts the minimum number of pods required to balance topology domains to within each constraint’s maxSkew.
This is useful for rebalancing pods across zones after a zone outage recovery.
Third-party components
List of third-party software used in the descheduler module:
-
Descheduler 0.37.0
License: Apache License 2.0
A tool to auto-fix scheduling issues Kubernetes does not cover itself.