Hands-on Guide on Kubernetes Resource Quotas & Limit Ranges

Kubernetes is a great orchestration platform, but it becomes a headache if it's not handled with proper resource management. If resources are not set up in the right way, it leads to performance issues or resource starvation. To maintain control of this, Kubernetes has the concept of Resource Quotas and Limit Ranges.

In this article, you’ll learn what these terms mean and how to use Resource Quota and Limit Ranges in Kubernetes.

What Are Resource Quotas?

A Resource Quota is a Kubernetes object that sets hard limits on the resources that can be consumed within a specific namespace. This helps control the total resource consumption and restricts how much CPU, memory, and other resources a namespace can request or consume.

Why Use Resource Quotas?

There are different reasons to use Resource Quotas. These are:

1. It prevents a single namespace from consuming all cluster resources.
2. It enables fair resource sharing among different teams or applications.
3. It ensures predictable cluster behavior by controlling resource usage.

Creating a Resource Quota

apiVersion: v1
kind: ResourceQuota
metadata:
  name: basic-quota
  namespace: mynamespace
spec:
  hard:
    pods: "10"
    requests.cpu: "4"
    requests.memory: "8Gi"
    limits.cpu: "8"
    limits.memory: "16Gi"

The above configuration sets quotas for CPU, memory, and the number of pods. In the given configuration, pods set the quota to 10 pods, requests.cpu and requests.memory set the total resource requests, limits.cpu and limits.memory define the maximum resource usage.

>> Take a look at Ultimate Guide to Resource Limits vs Requests

How Resource Quotas Work?

When a pod or object is created in a namespace with a Resource Quota, the Kubernetes API checks if the creation would exceed any quota. If it does, the request is denied.

Example:

If a namespace allows up to 4 pods, any attempt to create a 5th pod will fail:

apiVersion: v1
kind: ResourceQuota
metadata:
  name: pod-quota
  namespace: dev
spec:
  hard:
    pods: "4"

Apply the configuration:

kubectl apply -f pod-quota.yaml

Output:
resourcequota/pod-quota created

Define the deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  namespace: dev
spec:
  replicas: 5
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx

Apply the configuration:

kubectl apply -f nginx-deployment.yaml

Output:
deployment.apps/nginx-deployment created

kubectl get pods -n dev
NAME                                READY   STATUS    RESTARTS   AGE
nginx-deployment-676b6c5bbc-7rkvw   1/1     Running   0          2m29s
nginx-deployment-676b6c5bbc-c224b   1/1     Running   0          2m29s
nginx-deployment-676b6c5bbc-f4hbs   1/1     Running   0          2m29s
nginx-deployment-676b6c5bbc-r2tq2   1/1     Running   0          2m29s

As you can see from the output, only four pods are running in the "dev" namespace. This confirms that the Resource Quota, which limits the namespace to 4 pods, is being enforced. Although the Deployment was created with 5 replicas, the ReplicaSet could only successfully create 4 pods—any attempt to create the 5th pod was rejected due to the quota limits.

‍

Advanced Resource Quota Features

1. Scope-based Resource Quotas

You can restrict resource quotas to specific types of resources using scopes. For example, you can apply quotas to:

- Pods with no priority class

- Pods with a specific priority class

- Pods using cross-namespace affinity terms

- Resources related to specific storage classes

a. Quota for Best-Effort Pods

It limits the resources used by pods that do not specify resource requests or limits (Best-Effort pods).

apiVersion: v1
kind: ResourceQuota
metadata:
  name: best-effort-quota
  namespace: example-namespace
spec:
  hard:
    pods: "5"
  scopes:
  - BestEffort

With this configuration, only up to 5 Best-Effort pods (pods with no resource requests or limits) are allowed in the example-namespace.

kubectl apply -f best-effort-quota.yml

Output:
resourcequota/best-effort-quota created

Expected Output:

kubectl describe quota best-effort-quota -n example-namespace
Name:       best-effort-quota
Namespace:  example-namespace
Scopes:     BestEffort
 * Matches all pods that do not have resource requirements set. These pods have a best effort quality of service.
Resource  Used  Hard
--------  ----  ----
pods      0     5

b. Quota for Pods with a Specific Priority Class

It restricts pods with a specific priority class to ensure critical workloads have limited and controlled resource usage.

apiVersion: v1
kind: ResourceQuota
metadata:
  name: high-priority-quota
  namespace: example-namespace
spec:
  hard:
    pods: "2"
  scopeSelector:
    matchExpressions:
    - scopeName: PriorityClass
      operator: In
      values:
      - high

Please Apply:

kubectl apply -f high-priority-quota.yml

Output:
resourcequota/high-priority-quota created

Expected Output:

kubectl describe quota high-priority-quota -n example-namespace
Name:       high-priority-quota
Namespace:  example-namespace
Resource    Used  Hard
--------    ----  ----
pods        0     2

With this configuration, only 2 pods with the priority class high can be created in the namespace.

c. Quota to Disable Cross-Namespace Pod Affinity

Cross-Namespace Pod Affinity rule allows pods in one namespace to specify rules that depend on pods in another namespace. Disabling it prevents pods in a namespace from using cross-namespace affinity or anti-affinity.

apiVersion: v1
kind: ResourceQuota
metadata:
  name: disable-cross-affinity
  namespace: example-namespace
spec:
  hard:
    pods: "0"
  scopeSelector:
    matchExpressions:
    - scopeName: CrossNamespacePodAffinity
      operator: Exists

With this configuration, Pods in example-namespace cannot specify namespaceSelector or namespaces in their pod affinity terms.

Apply it:

apiVersion: v1
kind: ResourceQuota
metadata:
  name: disable-cross-affinity
  namespace: example-namespace
spec:
  hard:
    pods: "0"
  scopeSelector:
    matchExpressions:
    - scopeName: CrossNamespacePodAffinity
      operator: Exists

With this configuration, Pods in example-namespace cannot specify namespaceSelector or namespaces in their pod affinity terms.

Apply it:

kubectl apply -f disable-cross-affinity.yml
resourcequota/disable-cross-affinity created

Expected Output:

kubectl describe quota disable-cross-affinity -n example-namespace
Name:       disable-cross-affinity
Namespace:  example-namespace
Resource    Used  Hard
--------    ----  ----
pods        0     0

2. Configuring Quotas for Object Counts

Resource quotas can manage object counts(total number of one particular resource kind in the Kubernetes API) like ConfigMaps, Secrets, PersistentVolumeClaims (PVCs), and more.

Example: Limit ConfigMaps and Secrets

It restricts the number of ConfigMaps and Secrets created in a namespace.

apiVersion: v1
kind: ResourceQuota
metadata:
  name: object-counts
  namespace: example-namespace
spec:
  hard:
    configmaps: "10"
    secrets: "5"

With this configuration, You can create up to 10 ConfigMaps and 5 Secrets in example-namespace.

Apply:

kubectl apply -f object-counts.yml

Output:
resourcequota/object-counts created

Expected Output:

kubectl describe quota object-counts -n example-namespace
Name:       object-counts
Namespace:  example-namespace
Resource    Used  Hard
--------    ----  ----
configmaps  0     10
secrets     0     5

>> Take a look at Kubernetes Health Checks

3. Combining Multiple Scopes

You can combine multiple scopes to enforce advanced restrictions.

Example: Quota for Best-Effort Pods Using a Specific Storage Class

apiVersion: v1
kind: ResourceQuota
metadata:
  name: storage-quota
  namespace: example-namespace
spec:
  hard:
    requests.storage: "50Gi"
  scopeSelector:
    matchExpressions:
    - scopeName: BestEffort
      operator: Exists
    - scopeName: StorageClass
      operator: In
      values:
      - gold

With this configuration, Only Best-Effort pods using the gold storage class are limited to a total of 50Gi of storage requests in the namespace.

Apply:

kubectl apply -f storage-quota.yml

Output:
resourcequota/storage-quota created

Expected Output:

kubectl describe quota storage-quota -n example-namespace
Name:       storage-quota
Namespace:  example-namespace
Resource         Used  Hard
--------         ----  ----
requests.storage 0     50Gi

4. Priority Class Enforcement

You can restrict the total memory (or CPU) that pods with a specific PriorityClass may request. For example, if you want to ensure that all pods with the "high-memory" PriorityClass in the prod namespace collectively request no more than 8Gi of memory, you can use the following ResourceQuota:

apiVersion: v1
kind: ResourceQuota
metadata:
  name: high-memory-quota
  namespace: prod
spec:
  hard:
    requests.memory: "8Gi"
  scopeSelector:
    matchExpressions:
    - scopeName: PriorityClass
      operator: In
      values:
      - high-memory

ScopeSelector restricts the quota to pods with the "high-memory" PriorityClass; the hard limit allows only a collective 8Gi of memory requests in the prod namespace (exceeding which new pods are denied); this enforces CPU/memory management by preventing high-priority pods from over-consuming resources.

Apply and describe:

kubectl apply -f high-memory-quota.yml
kubectl describe quota high-memory-quota -n prod
Name:       high-memory-quota
Namespace:  prod
Resource         Used  Hard
--------         ----  ----
requests.memory  0     8Gi

5. Advanced Quota Adjustments with Controllers

Sometimes, you may need dynamic quotas based on cluster capacity or namespace needs. For example:

• Dynamic scaling: It proportionally distributes cluster resources among namespaces.
• Quota expansion: You can adjust namespace quotas as nodes are added to the cluster.

To implement this, you would write a custom Kubernetes controller that watches resource usage and adjusts hard limits dynamically based on signals like node addition or increased namespace demand.

‍

Now, let’s understand LimitRanges in K8s:

What Are Limit Ranges?

A LimitRange is a k8s object that enforces default or maximum resource limits on individual pods and containers in a namespace or makes sure that a single object cannot monopolize all available resources within a namespace.

Why Use Limit Ranges?

There are different reasons to use LimitRanges. These are:

- It prevents individual pods from requesting excessive resources.

- It enforces strict minimum and maximum resource requests and limits for pods, preventing any container from falling outside the defined boundaries.

- It provides default resource requests and limits (at container level) when not specified.

Creating and Using Limit Ranges

Example 1: Basic LimitRange

This LimitRange enforces default resource requests and limits for containers in a namespace.

apiVersion: v1
kind: LimitRange
metadata:
  name: basic-limits
  namespace: example-namespace
spec:
  limits:
  - default:
      cpu: "500m"
      memory: "256Mi"
    defaultRequest:
      cpu: "250m"
      memory: "128Mi"
    type: Container

This configuration sets the default Request to 250m CPU, 128Mi memory and default limit to 500m CPU, 256Mi memory.

Example 2: Enforcing Minimum and Maximum Resource Constraints

This LimitRange restricts the range of CPU and memory resources for containers.

apiVersion: v1
kind: LimitRange
metadata:
  name: range-limits
  namespace: example-namespace
spec:
  limits:
  - min:
      cpu: "100m"
      memory: "64Mi"
    max:
      cpu: "2"
      memory: "1Gi"
    type: Container

With the above configuration, Containers must specify resource requests/limits(Min: 100m CPU, 64Mi memory) and Max(2 CPU, 1Gi memory). If it attempts to create containers outside this range, it results in an error.

Example 3: Managing Pod Resource Limits

LimitRanges can also enforce total resource limits at the pod level.

apiVersion: v1
kind: LimitRange
metadata:
  name: pod-limits
  namespace: example-namespace
spec:
  limits:
  - max:
      cpu: "4"
      memory: "4Gi"
    type: Pod

With the above configuration, the total CPU and memory for a pod cannot exceed 4 cores and 4Gi memory. 

Apply it:

kubectl apply -f pod-limits.yml
kubectl describe limitrange pod-limits -n example-namespace

Name:       pod-limits
Namespace:  example-namespace
Type        Resource  Max
----        --------  ---
Pod         cpu       4
Pod         memory    4Gi

Now, create a pod without resource limits or requests:

apiVersion: v1
kind: Pod
metadata:
  name: pod-no-resources
  namespace: example-namespace
spec:
  containers:
  - name: demo
    image: busybox
    command: ["sleep", "3600"]

Apply:

kubectl apply -f pod-no-resources.yml

Output:
Error from server (Forbidden): error when creating "pod-no-resources.yml": pods "pod-no-resources" is forbidden: [maximum cpu usage per Pod is 4.  No limit is specified, maximum memory usage per Pod is 4Gi.  No limit is specified]

Since your Pod-level LimitRange (of type Pod) enforces that the total CPU and memory must be defined and within limits, the absence of these values causes the pod to be rejected.

Now, let's create a pod that explicitly requests more resources than allowed:

apiVersion: v1
kind: Pod
metadata:
  name: pod-out-of-range
  namespace: example-namespace
spec:
  containers:
  - name: demo
    image: busybox
    command: ["sleep", "3600"]
    resources:
      requests:
        cpu: "5"      # Exceeds the pod limit of 4 CPU
        memory: "5Gi" # Exceeds the pod limit of 4Gi memory
      limits:
        cpu: "5"
        memory: "5Gi"

Apply:

kubectl apply -f pod-out-of-range.yml

The pod creation will be rejected with an error message like:

Error from server (Forbidden): error when creating "pod-out-of-range.yml": pods "pod-out-of-range" is forbidden: [maximum cpu usage per Pod is 4, but limit is 5, maximum memory usage per Pod is 4Gi, but limit is 5Gi]

It clearly shows how LimitRanges enforcing pod-level limits rejects a pod whose total requested CPU and memory exceed the defined maximum.

Advanced Use Cases

1. Combining Multiple Limit Ranges

You can create multiple Limit Ranges to enforce different constraints in a namespace.

apiVersion: v1
kind: LimitRange
metadata:
  name: multi-limits
  namespace: example-namespace
spec:
  limits:
  - default:
      cpu: "1"
      memory: "512Mi"
    max:
      cpu: "2"
      memory: "1Gi"
    min:
      cpu: "500m"
      memory: "256Mi"
    type: Container

2. Applying Limits to Ephemeral Storage

You can manage ephemeral storage limits to prevent pods from consuming excessive disk space.

apiVersion: v1
kind: LimitRange
metadata:
  name: ephemeral-storage-limits
  namespace: example-namespace
spec:
  limits:
  - max:
      ephemeral-storage: "2Gi"
    default:
      ephemeral-storage: "1Gi"
    type: Container

With the above configuration, Containers’ default is 1Gi ephemeral storage and cannot exceed 2Gi ephemeral storage.

Using Resource Quotas and Limit Ranges Together

Using Resource Quotas and LimitRanges together is always a good practice, as it restricts total resource consumption in a namespace with a Resource Quota and ensures each pod and container in that namespace cannot monopolize all available resources within a namespace.

1. Define a Resource Quota:

apiVersion: v1
kind: ResourceQuota
metadata:
  name: namespace-quota
  namespace: mynamespace
spec:
  hard:
    requests.cpu: "4"
    requests.memory: "8Gi"
    limits.cpu: "8"
    limits.memory: "16Gi"

2. Define a Limit Range:

apiVersion: v1
kind: LimitRange
metadata:
  name: container-limits
  namespace: mynamespace
spec:
  limits:
  - type: Container
    max:
      cpu: "2"
      memory: "2Gi"
    min:
      cpu: "200m"
      memory: "256Mi"
    default:
      cpu: "500m"
      memory: "512Mi"

It together ensures the namespace has a total resource cap (Resource Quota) while enforcing per-container resource constraints and defaults (LimitRange) to prevent resource monopolization.

Best Practices

There are some best practices that you can follow:

1. To effectively manage resources in Kubernetes, it’s important to plan and implement ResourceQuotas and LimitRanges thoughtfully. Start by strategically planning quotas and limits based on your team or application’s specific resource needs. Avoid overly restrictive quotas that can stifle development while ensuring that resources are allocated fairly and efficiently across namespaces.

2. When using both ResourceQuotas and LimitRanges, combine them both for optimal resource management. Use Resource Quotas to set hard caps on total resource consumption within a namespace, such as CPU, memory, or object counts. Add this with LimitRanges to define policies at the Pod or Container level, ensuring that individual workloads do not monopolize resources or exceed sensible limits.

3. Monitoring and adjusting quotas regularly is important for maintaining balance as workloads evolve. When necessary, refine these limits to accommodate changing application demands or cluster capacity.

4. To avoid miscommunication or misconfiguration, document your resource policies clearly. Provide comprehensive guidelines for developers and administrators to ensure they understand how quotas and limits are applied and how they can design their workloads to comply with these constraints.

So, you have understood that resource management is important for running a stable Kubernetes environment. By implementing Resource Quotas, you can set hard limits on overall resource consumption per namespace, while Limit Ranges ensure that individual pods and containers follow defined constraints. 

Now that you understand how to use Resource Quotas and Limit Ranges, you can start applying them to your clusters and improve resource governance across your deployments!