K8s – Cordon and Drain CLI

In this blog, we will take a look at Cordon & Drain in Kubernetes.

• Cordon – Stops new pods from being admitted, however existing pods on it keep running ie: Host rules on DRS to keep new VM’s away
• Drain – moves all pods away, so this is the ESX maintainence mode with our DRS ie: Maintenance mode + Evacuate all VMs
• UnCordon – Readmit new pods to it but do note the existing pods wont move around so no load balancing

Check the nodes

 kubectl get nodes -o wide

Check the pods

 k get pods -A -o wide

Drain a Worker Node — move all pods away (like ESXi maintenance mode with DRS)

kubectl drain k3s-worker-01 --ignore-daemonsets --delete-emptydir-data

DaemonSet pods are special kubernertes resources that can’t be moved they’re designed to run on every node+

Check the nodes

kubectl get nodes -o wide

Check the pods so we can see pods running on worker01

kubectl get pods-o wide

The logic here is any future deployments will never hit this worker node once scheduling is disabled for the worker node. Use grep on above command on use field selector

 kubectl get pods -A --field-selector spec.nodeName=k3s-worker-01

Bring it back: Uncordon

Node becomes schedulable again. Existing pods do NOT automatically move back — only new pods will be scheduled there.

kubectl uncordon k3s-worker-01

Recheck the node status and pods running on it

kubectl get pods -A --field-selector spec.nodeName=k3s-worker-01

Node becomes schedulable again but kubernetes does not automatically rebalance running pods. Unlike DRS which continuously moves VMs, Kubernetes only schedules pods at creation time. Once running a pod stays on its node unless evicted and so to get DRS like features on Rancher we need to get Descheduler so even if a node goes down Kubernetes can reblance the scheduling to other nodes.

Keep node as unschedulable : Cordon

To just put node into an unschedulable state, we can just use CORDON which means no new pods can run on it

kubectl cordon k3s-worker-01