Scale Lab – Proxmox CLI Runbook

Shut down top-down (Scale, then nodes, then SANs) and start bottom-up (DNS, SANs, then nodes). The Scale nodes get their disks over iSCSI from the SAN VMs, so a SAN must never go down under a running node or start after one. All commands run on the Proxmox host pve unless noted.

DNS is Technitium at 172.16.11.5 and is not one of these VMs; pve itself resolves through it.

Networks

All lab NICs sit on vmbr1 (VLAN-aware). Odd VLAN = siteA, even = siteB; subnet 172.16.<last 2 digits of VLAN>.0/24, gateway .1 on pve. VLAN 1611 is native, so siteA mgmt NICs get no tag.

Check the bridge (1611 should show PVID Egress Untagged):

grep -A6 'iface vmbr1' /etc/network/interfaces
bridge vlan show dev vmbr1

Add a routed VLAN gateway on pve, in /etc/network/interfaces (isolated VLANs need no stanza):

auto vmbr1.16xx
iface vmbr1.16xx inet static
    address 172.16.xx.1/24

ifreload -a
ip -br a | grep vmbr1

Fix pve’s own DNS if /etc/resolv.conf is a dangling symlink:

rm /etc/resolv.conf
printf 'search lab.internal\nnameserver 172.16.11.5\n' > /etc/resolv.conf

Create VMs

Clone from template 9002 (scale-node-template-v2). Clones inherit the template’s root SSH key, so root SSH between nodes needs no setup.

qm clone 9002 115 --full --name sitea-ces01

Set NICs. siteA (mgmt untagged):

qm set 115 --net0 virtio,bridge=vmbr1 --net1 virtio,bridge=vmbr1,tag=1613 --net2 virtio,bridge=vmbr1,tag=1615

siteB:

qm set 125 --net0 virtio,bridge=vmbr1,tag=1612 --net1 virtio,bridge=vmbr1,tag=1614 --net2 virtio,bridge=vmbr1,tag=1616

Change a tag but keep the MAC:

qm config 115 | grep ^net0
qm set 115 --net0 virtio=BC:24:11:1B:E5:75,bridge=vmbr1

Verify:

for id in 115 116 125 126; do echo "== $id"; qm config $id | grep -E '^(name|memory|net[0-2]):'; done

Configure guests from pve

The QEMU guest agent (in the template) runs commands as root inside the VM, with no console or SSH. Check it first:

qm guest ping 115 && echo agent ok

Set hostname, machine-id, SSH host keys and static netplan, then reboot. One line per VM in the list; expect done and "exitcode" : 0:

# id hostname mgmt daemon ces mgw dgw other-site-daemon-subnet
while read -r ID H M D C MGW DGW OTHER; do
  echo ">> $ID $H"
  qm guest exec $ID --timeout 120 -- bash -c "
    hostnamectl set-hostname $H
    rm -f /etc/machine-id /var/lib/dbus/machine-id && systemd-machine-id-setup
    rm -f /etc/ssh/ssh_host_* && ssh-keygen -A
    rm -f /etc/netplan/*.yaml
    cat > /etc/netplan/50-lab.yaml <<EOF
network:
  version: 2
  ethernets:
    enp6s18:
      addresses: [$M/24]
      routes: [{to: default, via: $MGW}]
      nameservers: {addresses: [172.16.11.5], search: [lab.internal]}
    enp6s19:
      addresses: [$D/24]
      routes: [{to: $OTHER, via: $DGW}]
    enp6s20:
      addresses: [$C/24]
EOF
    chmod 600 /etc/netplan/50-lab.yaml
    sed -i '/127.0.1.1/d' /etc/hosts && echo '127.0.1.1 $H.lab.internal $H' >> /etc/hosts
    netplan generate && echo done
  " | grep -E '"out-data"|"exitcode"'
  qm reboot $ID
done <<'LIST'
115 sitea-ces01 172.16.11.15 172.16.13.15 172.16.15.15 172.16.11.1 172.16.13.1 172.16.14.0/24
116 sitea-ces02 172.16.11.16 172.16.13.16 172.16.15.16 172.16.11.1 172.16.13.1 172.16.14.0/24
125 siteb-ces01 172.16.12.25 172.16.14.25 172.16.16.25 172.16.12.1 172.16.14.1 172.16.13.0/24
126 siteb-ces02 172.16.12.26 172.16.14.26 172.16.16.26 172.16.12.1 172.16.14.1 172.16.13.0/24
LIST

Verify by IP from pve. A siteA mgmt IP failing while the rest pass means net0 still has tag=1611:

for ip in 172.16.11.15 172.16.13.15 172.16.15.15 172.16.11.16 172.16.13.16 172.16.15.16 172.16.12.25 172.16.14.25 172.16.16.25 172.16.12.26 172.16.14.26 172.16.16.26; do ping -c1 -W1 $ip >/dev/null && echo "$ip ok" || echo "$ip FAIL"; done

DNS records

Each Scale node gets an A + PTR record for <name> (mgmt IP) and <name>-d (daemon IP) in Technitium. Create an API token under Administration → Sessions → Create Token, and delete it when done.

read -rsp "Technitium API token: " TOKEN; echo
DNS=http://172.16.11.5:5380; ZONE=lab.internal
while read -r name ip; do
  curl -s "$DNS/api/zones/records/add?token=$TOKEN&zone=$ZONE&domain=$name.$ZONE&type=A&ipAddress=$ip&ptr=true&createPtrZone=true&overwrite=true" | grep -o '"status":"[a-z]*"' | sed "s/^/$name: /"
done <<'EOF'
sitea-ces01 172.16.11.15
sitea-ces01-d 172.16.13.15
sitea-ces02 172.16.11.16
sitea-ces02-d 172.16.13.16
siteb-ces01 172.16.12.25
siteb-ces01-d 172.16.14.25
siteb-ces02 172.16.12.26
siteb-ces02-d 172.16.14.26
EOF

Verify forward and reverse:

for h in sitea-ces01 sitea-ces01-d sitea-ces02 sitea-ces02-d siteb-ces01 siteb-ces01-d siteb-ces02 siteb-ces02-d; do printf '%-16s %s\n' $h "$(dig +short $h.lab.internal @172.16.11.5)"; done
dig +short -x 172.16.13.15 @172.16.11.5

Cross-site check from inside the cluster (as awk):

ssh awk@sitea-mgmt01 'for h in sitea-ces01-d sitea-ces02-d siteb-ces01-d siteb-ces02-d; do ping -c1 -W1 $h >/dev/null && echo "$h ok" || echo "$h FAIL"; done'

Shutdown

1. Stop Scale on each site (skip if GPFS is not running). Root SSH is disabled, so log in as awk:

ssh -t awk@sitea-mgmt01 sudo /usr/lpp/mmfs/bin/mmshutdown -a
ssh -t awk@siteb-mgmt01 sudo /usr/lpp/mmfs/bin/mmshutdown -a

2. Shut down the CES, NSD and mgmt nodes:

for id in 115 116 125 126 113 114 123 124 111 121; do echo "stopping $id"; qm shutdown $id --timeout 180; done

3. Shut down the SANs last:

for id in 119 129; do echo "stopping $id"; qm shutdown $id --timeout 180; done

4. Confirm nothing is still running (no output = all off), then power off the host:

qm list | grep running
poweroff

Startup

1. Power on pve, then check DNS answers. Expect 172.16.11.11; if not, start Technitium first:

dig +short sitea-mgmt01.lab.internal @172.16.11.5

2. Start the SANs first:

qm start 119; qm start 129

3. Wait about a minute for the iSCSI targets, then start the mgmt, NSD and CES nodes:

for id in 111 121 113 114 123 124 115 116 125 126; do echo "starting $id"; qm start $id; done

4. Confirm all 12 VMs show running:

qm list

5. Start Scale on each site and check node state (all active):

ssh -t awk@sitea-mgmt01 "sudo /usr/lpp/mmfs/bin/mmstartup -a && sudo /usr/lpp/mmfs/bin/mmgetstate -a"
ssh -t awk@siteb-mgmt01 "sudo /usr/lpp/mmfs/bin/mmstartup -a && sudo /usr/lpp/mmfs/bin/mmgetstate -a"