Part 4 – Deploying Spectrum Scale VM’s

This section focuses on building all the VMs we need from the Proxmox UI and attaching each VM to the SAN-based LUNs we created in the last chapter.

The topology we are building is some similar to this one, so each site gets three Scale nodes: one management node and two NSD servers.

Every node has three network cards: one for management, one for Scale’s own traffic (the daemon network), and one for the SAN.

Login to Proxmox UI as admin and click Datacenter

Go to Permissions > Pools > Create a new Site as Site B

In VMware we will make a VM folder. Proxmox’s equivalent is a Pool.

Deploy the first VM from the template. Right-click scale-node-template-v2 → Clone and fill in the details:

• Target Node – pve
• VM ID – 121
• VM Name – siteb-mgmt01
• resource pool -site B (same as VM folder)
• Mode – Choose full clone
• Datastore – 2 TB DS

At the bottom of the screen, a task runs

Edit the VM settings to adjust Memory to 4 GB and untick “Ballooning Device”.

Under Network settings, define three NIC cards

• net0: siteB mgmt (SSH, DNS)
  • Bridge vmbr1, VLAN tag 1612
  • mgmt port group
• net1: siteB daemon network (GPFS and AFM traffic)
  • Bridge vmbr1, VLAN Tag 1614
  • Model VirtIO, Firewall unticked
  • Scale daemon port group
• net2: siteB SAN traffic
  • Bridge vmbr1, VLAN Tag 1618
  • iSCSI port group

So each VM will have three networks now

Right click the VM and power it on and login as awk user via its IP

Change host name

root@scale-template:~# hostnamectl set-hostname siteb-mgmt01

root@scale-template:~# hostnamectl set-hostname siteb-mgmt01

Add all scale nodes into /etc/hosts file

Assign an IP address on each VLAN

cat > /etc/netplan/01-lab.yaml <<'EOF'
network:
  version: 2
  ethernets:
    enp6s18:
      addresses: [172.16.12.21/24]
      routes: [{to: default, via: 172.16.12.1}]
      nameservers: {addresses: [172.16.11.5], search: [lab.internal]}
    enp6s19:
      addresses: [172.16.14.21/24]
      routes: [{to: 172.16.13.0/24, via: 172.16.14.1}]
    enp6s20:
      addresses: [172.16.18.21/24]
EOF

cat > /etc/netplan/01-lab.yaml <<'EOF'
network:
  version: 2
  ethernets:
    enp6s18:
      addresses: [172.16.12.21/24]
      routes: [{to: default, via: 172.16.12.1}]
      nameservers: {addresses: [172.16.11.5], search: [lab.internal]}
    enp6s19:
      addresses: [172.16.14.21/24]
      routes: [{to: 172.16.13.0/24, via: 172.16.14.1}]
    enp6s20:
      addresses: [172.16.18.21/24]
EOF

Change the permissions to 600 so only root can read the file:

chmod 600 /etc/netplan/01-lab.yaml

Set the iSCSI initiator name

Every clone starts with the template’s random name. Replace it with the name the SAN’s ACL expects:

cat /etc/iscsi/initiatorname.iscsi

Reboot the VM and Summary → IPs should show 172.16.12.21, 172.16.14.21 and 172.16.18.21.

Connect to the SAN

Now SSH to the VM on its new address, 172.16.12.21, and switch to root.

Restart the iSCSI service using the command

 systemctl restart iscsid

To logon to our SAN we will run

iscsiadm -m node --login

log in to the SAN run

iscsiadm -m node --login

Run lsblk to see all our SAN based luns and we should 11 luns.

Repeat the same steps for the other two siteB nodes, changing the VM ID, hostname, IP addresses and initiator name each time:

Repeat for siteA as well, using the siteA VLANs: 1611 for mgmt (untagged on our bridge), 1613 for daemon and 1617 for SAN, with sitea-san’s portal at 172.16.17.19.

Passwordless SSH between the nodes on Site B

The template already carries the same SSH keys for both users on every node, so we only need each node to trust the others’ host keys. On each site’s mgmt node, do it once as a root and our user awk…

Establish a session to our management on each site ie : siteb-mgmt01 as root user and run this

root@siteb-mgmt01:~# for h in siteb-mgmt01 siteb-nsd01 siteb-nsd02; do ssh-keyscan $h; done > ~/.ssh/known_hosts 2>/dev/null

Test Password less Authentication for root user

root@siteb-mgmt01:~# for h in siteb-mgmt01 siteb-nsd01 siteb-nsd02; do ssh $h hostname; done
siteb-mgmt01
siteb-nsd01
siteb-nsd02

Likewise, establish a session to our management on each site ie : siteb-mgmt01 & sitea-mgmt01 as awk user and run this

awk@siteb-mgmt01:~$ for h in siteb-mgmt01 siteb-nsd01 siteb-nsd02 siteb-mgmt01-d siteb-nsd01-d siteb-nsd02-d; do ssh-keyscan $h; done > ~/.ssh/known_hosts 2>/dev/null
awk@siteb-mgmt01:~$ scp ~/.ssh/known_hosts siteb-nsd01:.ssh/
awk@siteb-mgmt01:~$ scp ~/.ssh/known_hosts siteb-nsd02:.ssh/

Test Password less Authentication for awk user

awk@siteb-mgmt01:~$ for h in siteb-mgmt01 siteb-nsd01 siteb-nsd02; do ssh $h hostname; done

Passwordless SSH between the nodes on Site A

Establish a session to our management on each site ie : sitea-mgmt01 as root user and run this

root@siteb-mgmt01:~# for h in siteb-mgmt01 siteb-nsd01 siteb-nsd02; do ssh-keyscan $h; done > ~/.ssh/known_hosts 2>/dev/null

Test Password less Authentication for root user

root@siteb-mgmt01:~# for h in siteb-mgmt01 siteb-nsd01 siteb-nsd02; do ssh $h hostname; done
siteb-mgmt01
siteb-nsd01
siteb-nsd02

Likewise, establish a session to our management on each site ie : siteb-mgmt01 as awk user . Each node sits on two Scale networks, and each name points to one of them:

awk@siteb-mgmt01:~$ for h in siteb-mgmt01 siteb-nsd01 siteb-nsd02 siteb-mgmt01-d siteb-nsd01-d siteb-nsd02-d; do ssh-keyscan $h; done > ~/.ssh/known_hosts 2>/dev/null
awk@siteb-mgmt01:~$ scp ~/.ssh/known_hosts siteb-nsd01:.ssh/
awk@siteb-mgmt01:~$ scp ~/.ssh/known_hosts siteb-nsd02:.ssh/

Test Password less Authentication for awk user

awk@siteb-mgmt01:~$ for h in siteb-mgmt01 siteb-nsd01 siteb-nsd02; do ssh $h hostname; done

That’s it and all are working so now we now have all six Scale nodes, each with its SAN disks and now we can begin the cluster creation.