With all the prerequisites out of the way, we finally now are going to start creating our cluster. We are going to use a tool called spectrumscale toolkit an IBM tool to build Storage Scale clusters and to do the configuration.
The toolkit is stored in path /usr/lpp/mmfs/5.1.8.2/ansible-toolkit/spectrumscale
Login to site A as root user and set the installer node and navigate to /usr/lpp/mmfs/5.1.8.2/ansible-toolkit/
Set the installer node
./spectrumscale setup -s 172.16.11.11The toolkit needs Ansible on the installer node, and it isn’t installed
Install python package on the Installer node
apt -y install python3-pip 
Install ansible package on the Installer node
pip3 install ansible==2.9.15
Rerun the setup again
./spectrumscale setup -s 172.16.11.11
Add the Nodes
An NSD server is a node that has SAN disks attached and serves them to the rest of the cluster over the network. In our design, sitea-mgmt01 serves the two metadata LUNs (LUN 0 and 1), and sitea-nsd01 and sitea-nsd02 serve the data LUNs.
Adds sitea-mgmt01 as a quorum, manager, admin and NSD server node.
./spectrumscale node add sitea-mgmt01 -q -m -a -n
Add sitea-nsd01 and sitea-nsd02 as quorum and NSD server nodes:
With three quorum nodes, the cluster stays up if any one of them fails. To show every node and its roles, run
./spectrumscale node list
Configure the cluster
Let’s create the cluster
./spectrumscale config gpfs -c sitea
Disable call home as its not needed for lab
./spectrumscale callhome disableRun a precheck
./spectrumscale install --precheck
Turn off Ubuntu’s periodic updates on all three nodes vi /etc/apt/apt.conf.d/10periodic
Set the ephemeral port range
GPFS opens extra network connections between nodes for some jobs, such as copying data and running commands. So, we will set in ephemeral port range between 60000 and 61000, so a firewall only needs that one range opened.
./spectrumscale config gpfs --ephemeral_port_range 60000-61000
Install GPFS
Then, back on sitea-mgmt01, run the install
./spectrumscale install
Validate the Cluster
Validate the cluster by running
mmlscluster
To check if GPFS is running on each node we run
mmgetstate -a
The four possible states are either
- active: GPFS is running and the node is part of the cluster.
- down: GPFS is stopped on that node.
- arbitrating: the node is starting and the cluster is agreeing on a manager (a few seconds).
- unknown: the node can’t be reached at all.
Move GPFS onto its own network
At the moment, everything GPFS does goes over the mgmt network (172.16.11.x) so we will now chnage GPFS to its own deamon network ( VLAN 1613) as per our diagram
Shutdown the cluster by running
mmshutdown -a
A node’s daemon address can only be changed using mmchnode so we change to -d
mmchnode --daemon-interface=sitea-mgmt01-d -N sitea-mgmt01
mmchnode --daemon-interface=sitea-nsd01-d -N sitea-nsd01
mmchnode --daemon-interface=sitea-nsd02-d -N sitea-nsd02
Validate by running mmlscluster
Start GPFS by running
mmstartup -aCheck Cluster status
Adding GUI access to Scale
The GUI is a separate add-on so for us to install the UI, we just need to pick a node that you want to be GUI node and this runs the webserver so lets add it to the toolkit plan
./spectrumscale node add sitea-mgmt01 -g
The GUI node is set so let’s commit the plan by running precheck and
./spectrumscale install --precheckIf there are no errors, install it
./spectrumscale install && ./spectrumscale deployLogin as https://172.16.11.11/
Add our user awk to Security admin
Login as awk
This is how we build a cluster with the spectrumscale toolkit. A cluster can also be built manually with a node file and the mm commands, and we’ll do that in the next chapter while setting up siteB.