Karpenter is a great way to cut your AWS bill. It provides a simple and flexible way to scale and optimize your resource consumption.

But when it comes to deploying DaemonSets, you need to take care of some specific configurations to ensure that your DaemonSet is well deployed. This guide will show you how to deploy a DaemonSet in a Karpenter context.

What is a DaemonSet?

A DaemonSet in Kubernetes is a specialized controller used to ensure that a copy of a particular pod runs on all nodes in a cluster. It is particularly useful for deploying background tasks or system-level services that need to run on every node, such as log collectors, monitoring agents, or network components.

Key features of DaemonSets include:

• Automatic Scheduling: When nodes are added to the cluster, the DaemonSet schedules the specified pod on the new nodes automatically.
• Cleanup: When nodes are removed, the DaemonSet cleans up the pods running on those nodes.

This makes DaemonSets a powerful tool for maintaining uniformity and reliability in the operation of essential services across a Kubernetes cluster.

Priority classes

There is a known issue with Karpenter and DaemonSets when scaling nodes. DaemonSets ensure a copy of a pod runs on every node, consuming additional resources that Karpenter does not consider, leading to potential resource contention and under-provisioned nodes.

This forces operators to over-provision their nodes, resulting in inefficient resource utilization and higher costs. While the Kubernetes community and Karpenter developers are working on solutions, users currently need to manually adjust resource allocations and monitor node utilization to mitigate these issues.

A way to resolve this problem is to use a PriorityClass and attach it to the DaemonSet we are creating.

What is a PriorityClass?

A PriorityClass in Kubernetes is a resource used to assign priority levels to pods. It helps the Kubernetes scheduler make decisions during resource contention by determining which pods should be scheduled first or evicted in case of resource shortages.

• Pods with higher priorities are scheduled before those with lower priorities.
• During resource shortages, lower-priority pods may be preempted (evicted) to free up resources for higher-priority pods.

This ensures that critical workloads have the resources they need to run effectively.

Use PriorityClass built-in Helm Charts

Many Helm charts include built-in values to automatically create and configure PriorityClasses. We have examples in our documentation:

• Kubernetes observability and monitoring with Datadog
• Deploy JupyterHub using Helm

Use Qovery's dedicated PriorityClass

When deploying Qovery on a cluster, a dedicated PriorityClass named qovery-standard-priority is created automatically. You can use this PriorityClass when deploying a new Helm chart or DaemonSet to ensure that DaemonSets deploy correctly, even during resource contention.

Targeting all the nodes

When deploying a DaemonSet, you can use taints and affinities to control where the DaemonSet pods are scheduled. This can help you ensure that the DaemonSet pods are deployed only on nodes that are interesting for you (for monitoring, logging etc..).

Nodepool default taints

By default, 2 nodepools are deployed with Karpenter: default and stable. On the stable nodepool, a taint has been defined to ensure that only pods having a toleration with the stable nodepool can be scheduled on it (key nodepool/stable).

How to target every node

To ensure that the DaemonSet pods are scheduled on every node, you can add a toleration to the DaemonSet pods that matches any taint. You also need to add an affinity to the DaemonSet pods to ensure that they are not scheduled on Fargate nodes.

Here's an example of how you can do this with a helm chart:

tolerations:
    - operator: Exists
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
          - matchExpressions:
            - key: eks.amazonaws.com/compute-type
              operator: NotIn
              values:
                - fargate