Part 1 – Building a Two-Site IBM Storage Scale Lab on Proxmox

In this blog, we are going to attempt to build a two-site IBM Storage Scale (GPFS) environment, with CES protocol nodes and AFM-based disaster recovery, on a single Proxmox host using IBM Storage Scale Developer Edition.

We will setup two independent clusters, shared SAN LUNs with two NSD servers each, a dedicated CES shared root, CES NFS, and AFM-DR between the sites

Lab Architecture

A NAT-ed lab network on Proxmox, isolated from the home LAN
A DNS zone with forward and reverse records for every lab host
sitea-san and siteb-san: iSCSI targets presenting 11 LUNs each, fenced to the NSD servers only
scale-template: Ubuntu 22.04 base VM ready to clone

I gave the proxmox API a try and it worked well so here is the command to create all these dns records quickly on technetium dns

API=http://localhost:5380/api
TOKEN=$(curl -s "$API/user/login?user=admin&pass=YOURPASS" | sed 's/.*"token":"\([^"]*\)".*/\1/')
echo $TOKEN

Create the zone and every record in one loop

curl -s "$API/zones/create?token=$TOKEN&zone=lab.internal&type=Primary"; echo
for r in sitea-mgmt01:11 sitea-mgmt02:12 sitea-nsd01:13 sitea-nsd02:14 sitea-ces01:15 sitea-ces02:16 \
         sitea-ces-vip:17 sitea-san:19 siteb-mgmt01:21 siteb-mgmt02:22 siteb-nsd01:23 siteb-nsd02:24 \
         siteb-ces01:25 siteb-ces02:26 siteb-ces-vip:27 siteb-san:29 client1:60 scale-template:10; do
  h=${r%%:*}; ip=172.16.11.${r##*:}
  curl -s "$API/zones/records/add?token=$TOKEN&domain=$h.lab.internal&zone=lab.internal&type=A&ipAddress=$ip&ptr=true&createPtrZone=true"; echo
done

Building the siteA SAN

Production Scale clusters on a SAN see the same LUNs from two NSD servers, a primary and a backup. To copy that in a lab, I gave each site a small Ubuntu VM running the Linux kernel’s iSCSI target. It presents every LUN to the four NSD servers of its site, and nothing else.

Just 2 vCPU, 4 GB RAM, a 40 GB OS disk and a separate 120 GB disk for the LUN files, on vmbr1 with the Proxmox firewall off and I will just mirror this for SiteB SAN.

JavaScript

qm create 119 --name sitea-san --memory 4096 --cores 2 --cpu host \
  --machine q35 --scsihw virtio-scsi-single \
  --scsi0 Local-2TB:40,cache=none,discard=on \
  --scsi1 Local-2TB:120,cache=none,discard=on \
  --net0 virtio,bridge=vmbr1,firewall=0 \
  --ide2 local:iso/ubuntu-22.04.1-live-server-amd64.iso,media=cdrom \
  --boot order='scsi0;ide2' --agent enabled=1 --ostype l26

qm create 119 --name sitea-san --memory 4096 --cores 2 --cpu host \
  --machine q35 --scsihw virtio-scsi-single \
  --scsi0 Local-2TB:40,cache=none,discard=on \
  --scsi1 Local-2TB:120,cache=none,discard=on \
  --net0 virtio,bridge=vmbr1,firewall=0 \
  --ide2 local:iso/ubuntu-22.04.1-live-server-amd64.iso,media=cdrom \
  --boot order='scsi0;ide2' --agent enabled=1 --ostype l26

Preparing the base Ubuntu VM template

Every Scale node is cloned from one template and in my lab I’ve got Storage_Scale_Developer-5.1.8.2-x86_64-Linux and also Storage_Scale_Developer-5.2.0.1-x86_64-Linux so we will test the upgrade later as the lab develops.

qm create 9000 --name scale-template --memory 8192 --cores 2 --cpu host \
  --machine q35 --scsihw virtio-scsi-single \
  --scsi0 Local-2TB:40,cache=none,discard=on \
  --net0 virtio,bridge=vmbr1,firewall=0 \
  --ide2 local:iso/ubuntu-22.04.1-live-server-amd64.iso,media=cdrom \
  --boot order='scsi0;ide2' --agent enabled=1 --ostype l26 --vga std

Then install the build prerequisite packages on the template vm

JavaScript

apt update
apt install -y open-iscsi lsscsi sg3-utils nfs-common \
  net-tools iputils-ping dnsutils traceroute tcpdump iperf3 ethtool curl wget rsync unzip \
  vim tmux htop iotop sysstat lsof jq bash-completion chrony qemu-guest-agent

apt update
apt install -y open-iscsi lsscsi sg3-utils nfs-common \
  net-tools iputils-ping dnsutils traceroute tcpdump iperf3 ethtool curl wget rsync unzip \
  vim tmux htop iotop sysstat lsof jq bash-completion chrony qemu-guest-agent