Taints and Tolerations

Taints and tolerations work together to control which pods can schedule on which nodes.

Taint is a mark on a node that says “stay away” so new pods will not schedule here unless they have a matching toleration. Existing pods already running on the node are not affected and keep running.

Toleration is a declaration on a pod that says “allow me on that node.”

In vSphere terms  imagine marking an ESXi host as a golden/restricted host in DRS. No new VMs land on it unless they have an explicit exception. VMs already running on that host stay running — they are not evacuated.

Taint a Node

# add taint
kubectl taint node k3s-worker-02 goldenhost=true:NoSchedule

# remove taint (add - at end)
kubectl taint node k3s-worker-02 goldenhost=true:NoSchedule-

# check taints on a node
kubectl describe node k3s-worker-02 | grep Taint

Three Taint Effects

Toleration on a Pod

Mark worker-02 as a restricted golden host so only specific pods allowed:

apiVersion: v1
kind: Pod
metadata:
  name: important
spec:
  containers:
    - name: important
      image: nginx
  tolerations:
    - key: goldenhost
      value: "true"
      effect: NoSchedule
  nodeSelector:
    kubernetes.io/hostname: k3s-worker-02

Any pod without the toleration will fail to schedule on worker-02:

0/3 nodes are available: 1 node(s) had untolerated taint(s)

The toleration must match the taint exactly and only pods that have this will be able to run on worker node 02

kubectl apply -f important.yaml
kubectl get pods -o wide

Output:

NAME        READY   STATUS    RESTARTS   AGE   IP           NODE
important   1/1     Running   0          6s    10.42.1.46   k3s-worker-02

Pod lands on worker-02 — toleration allowed it through the taint.

Toleration only allows a pod to run on a tainted node — it does not guarantee it will land there. The pod could still schedule on any other node.

To guarantee placement on a specific node use toleration + nodeSelector or nodeAffinity together:

spec:
  tolerations:           # allows entry to tainted node
    - key: goldenhost
      value: "true"
      effect: NoSchedule
  nodeSelector:          # pins it to that specific node
    kubernetes.io/hostname: k3s-worker-02

Quick Reference

# add taint
kubectl taint node <node> <key>=<value>:<effect>

# remove taint
kubectl taint node <node> <key>=<value>:<effect>-

# check node taints
kubectl describe node <node> | grep Taint

# check pod tolerations
kubectl describe pod <pod> | grep Toleration

# count pods per node
kubectl get pods -A -o wide | grep Running | awk '{print $8}' | sort | uniq -c