Setting Kubernetes CPU limits may do more harm than good.
Common knowledge seems to be that setting CPU limits for containers running on Kubernetes is an anti-pattern. As outlined in this excellent article by Shon Lev-Ran - in most cases CPU limits in Kubernetes do more harm than good.
Why Setting Kubernetes CPU Limits Makes Little Sense
A very basic explanation for this is that the Linux kernel scheduler is a CFS (a Completely Fair Scheduler) - which means that as long as a process has CPU requests defined it will get its share of CPU time. Even if your container goes crazy (because it doesn't have limits) and tries to occupy all of the time on all of the machine’s cores - the scheduler won’t let it do that. It will continue to share the CPU time in a completely fair way between all of the processes in the system according to what they request. Yes, if all the processes collectively use up all of the CPU time and there’s no more idle time to assign - they will eventually all get throttled. And that’s the only thing that k8s CPU limits protect us from - node CPU overbooking. And even that - only if all the processes on the node have CPU limits defined. Otherwise the limitless containers may still occupy all of the idle capacity and cause all the rest to get throttled.
The flip side of that is that a container that has CPU limits will get throttled even when there are spare CPU cycles on the system.
Bottomline is - if efficient and cost-effective resource utilization is your goal - don’t set CPU limits for containers running on Kubernetes!
Tim Thockin - one of Kubernetes creators got very clear about this in this famous X-reply (He also talks about setting memory limits equal to requests, but that’s a topic for another blog) :
Removing K8s CPU Limits - Success Stories from the Trenches
The positive effect on system performance and availability that’s stated by Thomas Peitz in the tweet above is something we’ve seen time after time with our customers.
Here are some examples:
- Paramount pictures mention significant reduction in system downtime after using PerfectScale to remove CPU limits on most of their workloads.
- DirecTV also used PerfectScale automation to remove CPU limits across the board - saving them multiple hours of manual work.
- Another thing many customers observe is that when removing CPU limits - memory consumption goes up. And here again PerfectScale automated pod right-sizing comes handy to adjust the memory requests for reliable operation.
Kubernetes CPU Limits in Real Life:
But as I said earlier - common knowledge isn’t all that common. Many engineers continue setting CPU limits on their containers. There are even code checkers and Kubernetes policy scanners that will warn you about K8s CPU limits not set. This is for example what the VSCode kubernetes add-on tells us::
Some folks even dare to go against the crowd and put up a fight for keeping the Kubernetes CPU limits in place
After all - if the CPU limits in Kubernetes made no sense - why would the Kubernetes developers even allow us to define them in the pod spec?
Kubernetes CPU Limits for Predictability
And as Shon outlines in his post - it’s widely known that Google engineers always set CPU limits on all of their production containers. With the reasoning being - they value predictability and consistency over performance (or cost)!
Another often stated reason for setting limits is performance testing - we usually want to do performance testing in predictable environments, without relying on idle capacity that may or may not be available.
And that, IMHO, makes a lot of sense - testing environments can’t and shouldn’t be configured the same as production ones. The goals and priorities are different - and that should be reflected in the configuration.
Bottom line - if predictable performance is of more value to your organization than optimal utilization or resource cost reduction - definitely keep your Kubernetes CPU limits in place (where it makes sense).
›› Take a look at the 8 Kubernetes cluster size best practices to establish and continually maintain the proper Kubernetes requests and limits.
PerfectScale approach to setting Kubernetes CPU Limits
At PefectScale our motto is “responsible Kubernetes cost optimization”. We do realize resource allocation has a direct impact on both system reliability and performance.
Our rule of thumb is: “Remove K8s CPU limits - they are inefficient, wasteful and may negatively impact performance”
And yet we are in the right-sizing business - which means we understand there’s no such thing as one-size-fits-all. PerfectScale automation is highly customizable and if predictability and consistency are your values (or are a better fit for a specific cluster or workload) - we definitely allow you to define CPU limits and provide the best values for these limits - based on actual CPU utilization and node configuration.
Ready to get your clusters optimized? Schedule a demo today with PerfectScale.
.png)
