Configure Rancher K8s – Cordon and Drain UI

In this blog, we will take a look at Cordon & Drain in Kubernetes via the Rancher UI

• Cordon – Stops new pods from being admitted, however existing pods on it keep running ie: Host rules on DRS to keep new VM’s away
• Drain – moves all pods away, so this is the ESX maintainence mode with our DRS ie: Maintenance mode + Evacuate all VMs
• UnCordon – Readmit new pods to it but do note the existing pods wont move around so no load balancing

We will use the UI fully to this task but the equivalent CLI’s are also copied .

To check the nodes, go to Cluster > Nodes

 kubectl get nodes -o wide

To check the pods, select the node worker-02 and we can see all pods running on it

kubectl get pods -A --field-selector spec.nodeName=k3s-worker-02

Drain a Worker Node — move all pods away (like ESXi maintenance mode with DRS)

kubectl drain k3s-worker-01 --ignore-daemonsets --delete-emptydir-data

DaemonSet pods are special kubernertes resources that can’t be moved they’re designed to run on every node+

Select the worker node 02 and hit drain

Rancher assumes --ignore-daemonsets always — because there’s no practical reason to ever not ignore them.

The pods are being drained

App with 1 replica: → drain kills it → gap → new one starts so there is downtime however for replicas=3 there is no downtime.

Check the nodes to see if its drained

kubectl get nodes -o wide

Check the pods so we can see pods running on worker01

kubectl get pods-o wide

The logic here is any future deployments will never hit this worker node once scheduling is disabled for the worker node. Use grep on above command on use field selector

 kubectl get pods -A --field-selector spec.nodeName=k3s-worker-01

Bring it back: Uncordon

Node becomes schedulable again. Existing pods do NOT automatically move back — only new pods will be scheduled there.

kubectl uncordon k3s-worker-02

Recheck the node status and pods running on it

kubectl get pods -A --field-selector spec.nodeName=k3s-worker-02

Node becomes schedulable again but kubernetes does not automatically rebalance running pods. When we uncordoned worker-02, new pods started going there as they were created or restarted but unlike DRS which continuously moves VMs, Kubernetes only schedules pods at creation time.

For Kubernetes to rebalance the scheduling to other nodes we need Descheduler