Have you ever faced the challenge of managing resource contention in a Kubernetes cluster? Critical applications slowing down as less important workloads consume valuable CPU and memory is a nightmare scenario for any DevOps engineer. Fortunately, there's a way to ensure your essential services always get the resources they need.
![](https://cdn.prod.website-files.com/635e4ccf77408db6bd802ae6/66bdca1e417f66a31d458668_66bdc9bc0d0ca24b08d21701_preemptible-pod-2.png)
In this blog, we'll explore preemptible pods – a powerful feature in Kubernetes that allows you to optimize node utilization by prioritizing critical workloads and preempting less important ones. We'll delve into the basics of pod priority and preemption, provide step-by-step instructions on setting up PriorityClasses, and walk through a practical example illustrating how to implement and observe pod preemption in action.
By the end of this post, you’ll understand how leveraging pod priority and preemption can enhance your cluster's efficiency and reliability, ensuring your critical applications always come first. Let’s dive in and discover how to keep your Kubernetes cluster running smoothly, even under heavy load.
What and Why Pod Priority and Preemption Matter
Pod priority is a mechanism that lets you define the importance of a pod relative to others. By assigning priority classes, you can control which pods should be scheduled first when resources are limited. While, preemption is the process where higher-priority pods can evict lower-priority pods to obtain the resources they need. This ensures that the most important workloads are always running.
But why Priority and Preemption?
Priority and Preemption are chosen above others strategies of kubernetes node utilization for specific scenarios where:
- Critical Workloads:
- Ensuring that high-priority workloads always have the necessary resources, even at the expense of less critical ones.
- Vital for applications with strict SLAs or those that are mission-critical.
- Resource Scarcity:
- Effective in environments with resource constraints where optimal resource utilization is crucial.
- Prevents resource starvation for important services by evicting lower-priority pods.
- Dynamic Workload Management:
- Ideal for clusters with highly dynamic workloads where the priority of applications can change frequently.
- Allows for a more responsive and adaptable resource allocation strategy.
While Priority and Preemption are powerful tools for managing node utilization and ensuring that critical workloads have the resources they need, there are several other strategies to optimize Kubernetes node utilization. Each approach has its own benefits and use cases, and choosing the right one depends on your specific requirements and environment. Like; Horizontal Pod Autoscaling (HPA), Vertical Pod Autoscaling (VPA), Cluster Autoscaling and Resource Quotas and Limits.
Walkthrough of a practical example
Step 1. Setting Up PriorityClasses - Define PriorityClasses
Let's create two PriorityClasses: one for high-priority pods and one for low-priority pods.
- high-priority: Used for critical pods.
- low-priority: Used for less critical pods.
Apply these PriorityClasses:
Step 2. Creating Pods with PriorityClasses
Now, let's create two pods, one with each PriorityClass.
High-Priority Pod:
Low-Priority Pod:
Apply these pods:
Step 3. Simulating Preemption
To see preemption in action, we need to simulate a scenario where the cluster is under resource pressure.
In this setup, we will use a Kubernetes cluster with one control plane and one worker node. This environment is ideal for this learning. Ensure your system has sufficient resources to run both the control plane and worker node. Typically, this would involve at least 2GB of RAM and 2 CPU cores.
Alternatively, you can follow along using the free Kubernetes playground provided by Killercoda.
- Create Resource Constraints:
- Deploy a workload that consumes significant resources.
Apply this deployment:
2. Deploy another High-Priority Pod:
- This will trigger preemption as the cluster is already under resource pressure.
Apply this pod:
Step 3. Observing Preemption and Pod Eviction
To observe the preemption:
![](https://cdn.prod.website-files.com/635e4ccf77408db6bd802ae6/66bcc3b2bfa4b0c1611e21e3_AD_4nXctQY-M7R3isCiYgzlAa0EzCoJ95sMxjBXkCoK3rKP0C4SRYi_laZ6eA7X9D-nDFbXoeeMnALcc3yvQ-mJ1UhkSzjjftv9w7QOnQ87tIGS48ZO_18lHFxMd_NAvLjsRZvLAjqd2nc0ieKgih529bsxLEK8B.png)
You should see that the low-priority-pod gets evicted to make room for the new-high-priority-pod.
How Candidates for Eviction Are Chosen
Kubernetes uses several factors to determine which pods to evict:
- Priority: Pods with lower priority are considered first.
- Quality of Service (QoS):
- Guaranteed: Pods with requests and limits set for all containers.
- Burstable: Pods with requests and limits set for some containers.
- BestEffort: Pods without any resource requests or limits.
Guaranteed pods are the most stable and are evicted last, while BestEffort pods are the least stable and are evicted first. In our example, if both high-priority-pod and low-priority-pod are of BestEffort QoS, the low-priority-pod gets evicted due to its lower priority.
Impact of Pod Eviction on Kubernetes Node Utilization
Pod eviction plays a crucial role in optimizing Kubernetes node utilization. By preempting lower-priority pods, Kubernetes ensures that high-priority applications have the necessary resources to run smoothly. This mechanism prevents critical services from being starved of resources, thus maintaining the overall performance and reliability of the cluster.
![](https://cdn.prod.website-files.com/635e4ccf77408db6bd802ae6/66bdca1e417f66a31d458671_66bdc9da65d8b29490790d1e_preemptible-pod-1.png)
Moreover, pod eviction helps in efficient resource management by reclaiming resources from non-essential or less important workloads. This proactive approach maximizes the use of available resources, reduces wastage, and maintains a balanced load across the nodes. Consequently, Kubernetes can handle varying workloads more effectively, ensuring that the most important applications remain operational even during peak demand.
Conclusion and Best Practises
Implementing pod priority and preemption in Kubernetes is a powerful way to ensure that your critical applications always receive the resources they need, enhancing the overall efficiency and reliability of your cluster. By prioritizing workloads effectively, you can maintain optimal performance and prevent resource contention issues that could disrupt essential services.
To make the most out of these features, regularly review and adjust your PriorityClasses based on changing workload demands. Monitor your cluster's performance to identify and resolve any potential bottlenecks. By following these best practices, you can leverage Kubernetes' advanced scheduling capabilities to keep your infrastructure resilient and responsive, even under heavy load.
![](https://cdn.prod.website-files.com/635e4ccf77408db6bd802ae6/66bdca1e417f66a31d45867d_66bdca1343b66ccc8c0bad22_K8S_preemptible_pods.gif)
![PerfectScale Lettermark](https://cdn.prod.website-files.com/634ebd5aab79d1fa815c5394/653e2cf7ae6bc5934456d4ab_CTA%20Logo.png)