Part 10 – Powering the Scale Lab Up and Down

This post covers the order that works, and the health checks I run afterwards so I know the lab is really back, 

DNS is Technitium zone lab.internal and everything below runs on the Proxmox host pve

Powerup Sequence

“Running” in qm list only means the VM has power. These checks go up the stack, and each layer depends on the one below, so when something fails, stop and fix it before moving on.

Run the Scale checks as root on each site’s mgmt01 (ssh awk@sitea-mgmt01, then sudo -i).

Layer 1: Proxmox host

qm list                       # all 12 lab VMs running
pvesm status                  # storage pools active, not fullbridge 
vlan show dev vmbr1           # VLAN-aware bridge, 1611 as native (PVID)

Layer 2: Network

Ping every node on every network from pve.

for ip in 172.16.11.11 172.16.11.13 172.16.11.14 172.16.11.15 172.16.11.16 172.16.11.19 172.16.12.21 172.16.12.23 172.16.12.24 172.16.12.25 172.16.12.26 172.16.12.29 172.16.13.11 172.16.13.13 172.16.13.14 172.16.13.15 172.16.13.16 172.16.14.21 172.16.14.23 172.16.14.24 172.16.14.25 172.16.14.26; do ping -c1 -W1 $ip >/dev/null && echo "$ip ok" || echo "$ip FAIL"done

Cross-site daemon route (1613 to 1614), from inside a cluster:

ssh awk@sitea-mgmt01 'for h in siteb-mgmt01-d siteb-nsd01-d siteb-ces01-d; do ping -c1 -W1 $h >/dev/null && echo "$h ok" || echo "$h FAIL"; done'

Layer 3: DNS (forward and reverse)

GPFS and CES both check names. CES won’t even add a floating IP without a reverse record.

for h in sitea-ces01 sitea-ces01-d siteb-ces01 siteb-ces01-d sitea-nfs01 siteb-nfs01; do  printf '%-16s %s\n' $h "$(dig +short $h.lab.internal @172.16.11.5)"donedig +short -x 172.16.15.100 @172.16.11.5

Layer 4: Time

Scale nodes need their clocks in sync. Lease and token timeouts get unhappy when they drift.

for h in sitea-mgmt01 sitea-nsd01 sitea-ces01 siteb-mgmt01 siteb-nsd01 siteb-ces01; do  printf '%-14s ' $h; ssh awk@$h 'timedatectl show -p NTPSynchronized --value'done

Layer 5: SAN and iSCSI

On a node, check it has logged in to its SAN and can see the LUNs:

iscsiadm -m session          # one session to the site's SAN 
portallsblk | grep -c disk   # roughly 12 disks: the OS disk + 11 LUNs

On the SAN itself (sitea-san / siteb-san), check the target and its LUNs:

targetcli ls /iscsi

Layer 6: GPFS cluster

mmgetstate -a

Every node active. arbitrating or down means quorum or network trouble.

mmgetstate -s

The summary shows quorum nodes needed vs. active. You want all quorum nodes active.

mmlscluster

Check the daemon node name column: every node, including the CES nodes, should be on its -d name (172.16.13.x / 172.16.14.x), not the mgmt network.

Layer 7: Disks and file systems

mmlsnsd -M                       # each NSD mapped to a device on its servers
mmlsdisk sitea_prod_scale_fs -e  # lists only disks NOT up/ready; no output = good
mmlsdisk cesSharedRoot -e
mmlsmount all -L                 # which nodes have each file system mounted

On siteB, swap in siteB’s file system name (mmlsfs all -T lists them). cesSharedRoot must be mounted on both CES nodes, or CES won’t start.

Layer 8: Overall health

mmhealth cluster show

The one-screen summary for the whole cluster. Drill into any node that isn’t HEALTHY:

mmhealth node show -N sitea-ces01.lab.internal

Layer 9: CES and NFS

mmces address list
mmces service list -a
mmces state show -a

• One floating IP per CES node (.100 and .101).
• NFS is running on both.

If both IPs are stuck on one node, check the policy and rebalance:

mmces address policy
mmces address move --rebalance

Run a health check

mmgetstate -a; mmhealth cluster show; mmces address list; mmces state show -a

Power-off sequence

1. Stop Scale on both sites

ssh -t awk@sitea-mgmt01 sudo /usr/lpp/mmfs/bin/mmshutdown -a
ssh -t awk@siteb-mgmt01 sudo /usr/lpp/mmfs/bin/mmshutdown -a

This unmounts every file system, stops CES (the floating IPs go down) and stops GPFS on every node.

2. Shut down the Scale nodes

CES first, then NSD, then mgmt:

for id in 115 116 125 126 113 114 123 124 111 121; do echo "stopping $id"; qm shutdown $id --timeout 180; done

qm shutdown asks the guest OS to shut down cleanly through ACPI and the guest agent. --timeout 180 gives each VM up to 3 minutes.

3. Shut down the SANs

for id in 119 129; do echo "stopping $id"; qm shutdown $id --timeout 180; done

4. Confirm, then power off the host

qm list | grep running    # no output = every VM is off
poweroff