Karpenter is a great way to cut your AWS bill. It provides a simple and flexible way to scale and optimize your resource consumption. But there is a known issue with capacity planning when deploying DaemonSets. In this guide, we will explore this issue and demonstrate how to resolve it using PriorityClass.
What is a DaemonSet?
A DaemonSet in Kubernetes is a specialized controller used to ensure that a copy of a particular pod runs on all nodes in a cluster. It is particularly useful for deploying background tasks or system-level services that need to run on every node, such as log collectors, monitoring agents, or network components.
Key features of DaemonSets include:
- Automatic Scheduling: When nodes are added to the cluster, the DaemonSet schedules the specified pod on the new nodes automatically.
- Cleanup: When nodes are removed, the DaemonSet cleans up the pods running on those nodes.
This makes DaemonSets a powerful tool for maintaining uniformity and reliability in the operation of essential services across a Kubernetes cluster.
What is the problem?
There is a known issue with Karpenter and DaemonSets when scaling nodes. DaemonSets ensure a copy of a pod runs on every node, consuming additional resources that Karpenter does not consider, leading to potential resource contention and under-provisioned nodes.
This forces operators to over-provision their nodes, resulting in inefficient resource utilization and higher costs. While the Kubernetes community and Karpenter developers are working on solutions, users currently need to manually adjust resource allocations and monitor node utilization to mitigate these issues.
A way to resolve this problem is to use a PriorityClass and attach it to the DaemonSet we are creating.
How to resolve it?
What is a PriorityClass?
A PriorityClass in Kubernetes is a resource used to assign priority levels to pods. It helps the Kubernetes scheduler make decisions during resource contention by determining which pods should be scheduled first or evicted in case of resource shortages.
- Pods with higher priorities are scheduled before those with lower priorities.
- During resource shortages, lower-priority pods may be preempted (evicted) to free up resources for higher-priority pods.
This ensures that critical workloads have the resources they need to run effectively.
Use PriorityClass built-in Helm Charts
Many Helm charts include built-in values to automatically create and configure PriorityClasses. We have examples in our documentation:
Use Qovery's dedicated PriorityClass
When deploying Qovery on a cluster, a dedicated PriorityClass named qovery-standard-priority is created automatically.
You can use this PriorityClass when deploying a new Helm chart or DaemonSet to ensure that DaemonSets deploy correctly, even during resource contention.
Conclusion
While tools like Karpenter are excellent for optimizing costs and reducing your AWS bill, some manual configuration is sometimes required. This issue demonstrates that adding a single PriorityClass can effectively prevent complex resource allocation problems.