With Ceph in place, my nested cluster finally has shared storage.
In vSphere, once you turn on HA and DRS for a cluster, they apply to every VM in it by default. If a host crashes, all its VMs restart elsewhere, and DRS balances all of them. You opt individual VMs out with VM overrides.
Proxmox works the other way round: it’s opt-in. HA only applies to the VMs you add as HA resources. When a node crashes:
- HA VMs are restarted on a surviving node, once HA has confirmed the failed node is fenced.
- Non-HA VMs stay down until their node comes back.
The same goes for load balancing. Proxmox’s automatic rebalancing (its DRS-like feature) and affinity rules only act on HA resources too. It’s designed so you pick exactly which workloads get the fencing and restart behaviour, and test VMs stay out of it.
- Recovery takes roughly 2 minutes because HA waits for the fence. That’s slower than vSphere HA, but it guarantees no split brain.
- Arm / Disarm HA (9.2) = cluster-wide on/off. Freeze keeps HA in charge but inactive; Ignore releases the VMs to manual control.
vMotion or Live Migration
Similar to that of our vSphere vMotiond, live migration is just movement of VM’s over a dedicated network and for us this is on VLAN 1612
Choose the HDD to move local Storage to Ceph
Storage vMotion is in the GUI too: VM → Hardware → Hard Disk → Disk Action → Move Storage,
Move to chosen storage
From the CLI, qm migrate runs on the node the VM is currently on:
root@pve-n2:~# qm list
VMID NAME STATUS MEM(MB) BOOTDISK(GB) PID
102 vm02 running 1024 8.00 2100
root@pve-n2:~#
root@pve-n2:~# qm migrate 102 pve-n1 --online --with-local-disks
Requesting HA migration for VM 102 to node pve-n1
root@pve-n2:~#
root@pve-n1:~# qm list
VMID NAME STATUS MEM(MB) BOOTDISK(GB) PID
102 vm02 running 1024 8.00 44359
9000 debian13-cloud stopped 1024 3.00 0
root@pve-n1:~#
Maintenance mode
This took me a while to figure out; there is no right click turn on maintenance mode saw in vSphere so this needs to be done via CLI. The GUI only shows a wrench icon once the commands are executed
Maintenance mode only evacuates HA VMs. Non-HA VMs keeps running on the node and will go down if it’s rebooted. Bulk Migrate them first or make them HA.
The current state of cluster has VM 101-109 added to a HA group however VM 110-112 is not a member of HA group so in this HA configured VM; will move away but the non-HA one stays for us to manually move it so let’s test it.
Enter maintenance
Login to the node that needs to be taken down for maintenance and run qm list to display all VM’s on it.
root@pve-n2:# qm list
102 vm02 running 1024 8.00 104231
104 vm04 running 1024 8.00 106589
107 vm07 running 1024 8.00 106532
112 vm12 running 1024 8.00 108498
root@pve-n2:#
The above output has 4 VM’s on it and this VM 112 is not part of an HA group so let’s trigger the maintenance event.
Ensure accessibility
Tells Ceph (storage) not to mark them out or start rebuilding set an OSD flag
c
root@pve-n2:# ceph osd set-group noout pve-n2
root@pve-n2:#
root@pve-n2:~#
noout is the flag we use to put a marker on Ceph to say pve-n2’s disks are going away for a bit. It’s planned, so don’t panic and copy data around so in short this is same vSAN maintenance mode with “Ensure accessibility”.
Enter Maintenance Mode
This is the command to drain the host
root@pve-n2:~# ha-manager crm-command node-maintenance enable pve-n2
Host has a wrench on it and its in maintenance mode. All VM’ except VM 112 has moved off from and this VM didnt move because it was not in a HA group
Manually move the VM using qm migrate <VMName> <Node name to move to> –online
root@pve-n2:# qm migrate 112 pve-n1 –online
2026-10-21 21:00:00 conntrack state migration not supported or disabled, active connections might get dropped
2026-10-21 21:00:00 use dedicated network address for sending migration traffic (172.16.12.41)
2026-10-21 21:00:00 starting migration of VM 112 to node ‘pve-n1’ (172.16.12.41)
2026-10-21 21:00:00 starting VM 112 on remote node ‘pve-n1’
2026-10-21 21:00:03 start remote tunnel
2026-10-21 21:00:04 ssh tunnel ver 1
2026-10-21 21:00:04 starting online/live migration on unix:/run/qemu-server/112.migrate
2026-10-21 21:00:04 set migration capabilities
2026-10-21 21:00:04 migration downtime limit: 100 ms
2026-10-21 21:00:04 migration cachesize: 128.0 MiB
2026-10-21 21:00:04 set migration parameters
2026-10-21 21:00:04 start migrate command to unix:/run/qemu-server/112.migrate
2026-10-21 21:00:05 migration active, transferred 391.1 MiB of 1.0 GiB VM-state, 466.4 MiB/s
2026-10-21 21:00:06 average migration speed: 512.4 MiB/s – downtime 58 ms
2026-10-21 21:00:06 migration completed, transferred 582.5 MiB VM-state
2026-10-21 21:00:06 migration status: completed
2026-10-21 21:00:10 migration finished successfully (duration 00:00:11)
root@pve-n2:#
The VM has been moved off so now we can just shut down the host with a power off
Power off the host
root@pve-n2:# poweroff
root@pve-n2:# Read from remote host 172.16.11.42: Connection reset by peer
Connection to 172.16.11.42 closed.
client_loop: send disconnect: Broken pipe
root@pve:~#
Ceph reports the missing disk
Exit maintenance
Enter the following command
root@pve-n2:# ha-manager crm-command node-maintenance disable pve-n2
root@pve-n2:# ceph osd unset-group noout pve-n2
root@pve-n2:~#
After few seconds, we can VM’s back on the host
Check the status of ceph by running ceph status
root@pve-n2:~# ceph status
cluster:
id: 63400913-953c-4b9f-bee6-f6bbb259245a
health: HEALTH_OK
services:
mon: 3 daemons, quorum pve-n1,pve-n2,pve-n3 (age 3m) [leader: pve-n1]
mgr: pve-n4(active, since 5m), standbys: pve-n3, pve-n1, pve-n2
osd: 4 osds: 4 up (since 3m), 4 in (since 14h)
data:
pools: 2 pools, 33 pgs
objects: 4.18k objects, 15 GiB
usage: 41 GiB used, 215 GiB / 256 GiB avail
pgs: 33 active+clean
io:
client: 0 B/s rd, 5.5 KiB/s wr, 0 op/s rd, 1 op/s wr