We will attempt to powerup a kubernetes cluster runing on VMware Workstation linux that was accidentally powered off and bring back all the pods.

Lab Hardware

VMware Workstation vmnet8 (NAT) configured as 172.16.77.0/24.

Powerup the VM’s in workstation

vmrun -T ws start ~/path to vmx file

vmrun -T ws start ~/vms/k3s-cp-01/k3s-cp-01/k3s-cp-01.vmx nogui

Check if all VM’s are running

vmrun list

Check if kubernetes nodes are up

 k get nodes -A

Check pods

 k get pods -A

Troubleshoot Pods

The VM running were forcefully powered off (not graceful shutdown) so pods never got the detach signal. RWO volumes don’t auto-release on ungraceful node loss until the 300s node.kubernetes.io/unreachable taint clears so one of my kuma pods is not starting so lets fix that

Run the get pods describe command on kuma to see what went wrong

k describe pod -n monitoring uptime-kuma-77d7fd658c-g4nhl

I use longhorn storage PVC’s here so this node was previously running on some other worker node and this as it restarted it probably would have been moved around to another worker. The ouput here shows that

To check my PVC attached to the kuma pod i’ll just run get pvc on its namespace

 k get pvc -n monitoring

Check where the volume was attached previously

 k get volumeattachment | grep pvc-29bb0d0a-6677-43c9-9a45-6cbb92d71741

So this now shows the pod was earlier on worker-02 and this time it restarted on cp-01

Check whats attached

 k describe pvc uptime-kuma-pvc -n monitoring

Delete ONLY the stale one (attached=true on wrong node)

kubectl delete volumeattachment <stale-one>

Check pods

k get pods -n monitoring