For anyone coming from NetApp, CES nodes are the filer heads, and the VIP is like a LIF (the floating data IP).
I’ve stood up 2 VM; s at each site which is going to act as as our ces nodes., each VM has 3 NIC; s on it. mgmt, Scale daemon, and CES, NFS,SMB traffic on it and we need the CES nodes to run on the daemon network 172.16.13.X
In this blog we add the CES nodes (protocol nodes) with the Spectrum Scale install toolkit and we will attempt to do a failover/failback of the LIF interface
| vm | mgmt | daemon | ces |
|---|---|---|---|
| sitea-ces01 | 172.16.11.15 | 172.16.13.15 | 172.16.15.15 |
| sitea-ces02 | 172.16.11.16 | 172.16.13.16 | 172.16.15.16 |
| siteb-ces01 | 172.16.12.25 | 172.16.14.25 | 172.16.16.25 |
| siteb-ces02 | 172.16.12.26 | 172.16.14.26 | 172.16.16.26 |
Deploy all the VM’s and attach 3 NIC’s on it as per above VLAN
Still as root on sitea-mgmt01, check that root can SSH to the CES nodes without a password. The key should already be there from the template
ssh -o BatchMode=yes -o StrictHostKeyChecking=accept-new root@sitea-ces01 hostname
ssh -o BatchMode=yes -o StrictHostKeyChecking=accept-new root@sitea-ces02 hostname
Each CES node must also know the host key of every node so login to sitea-ces01.
ssh root@sitea-ces01
ssh -o StrictHostKeyChecking=accept-new sitea-mgmt01.lab.internal hostname
ssh -o StrictHostKeyChecking=accept-new sitea-nsd01.lab.internal hostname
ssh -o StrictHostKeyChecking=accept-new sitea-nsd02.lab.internal hostname
ssh -o StrictHostKeyChecking=accept-new sitea-ces01.lab.internal hostname
ssh -o StrictHostKeyChecking=accept-new sitea-ces02.lab.internal hostnameEach CES node must also know the host key of every node so login to sitea-ces02 a well.
ssh root@sitea-ces02
ssh -o StrictHostKeyChecking=accept-new sitea-mgmt01.lab.internal hostname
ssh -o StrictHostKeyChecking=accept-new sitea-nsd01.lab.internal hostname
ssh -o StrictHostKeyChecking=accept-new sitea-nsd02.lab.internal hostname
ssh -o StrictHostKeyChecking=accept-new sitea-ces01.lab.internal hostname
ssh -o StrictHostKeyChecking=accept-new sitea-ces02.lab.internal hostnameAdding protocol nodes
For adding protocol nodes via spectrum scale took kit we use -p option
- -p = Protocol node (CES: NFS, SMB, S3, floating IPs)
- -q = Quorum node
- -m =Manager node
- -n =NSD server
- -q = admin node
- -g -gui server
Run ./spectrumscale node list to check current status of all nodes.
For adding protocol nodes via spectrum scale took kit we use -p option and add both our nodes
./spectrumscale node add sitea-ces01.lab.internal -p
./spectrumscale node add sitea-ces02.lab.internal -p
Configure CES
Now we will use our cesSharedroot filesystem we created earlier. This small partition holds the info about protocol node mounts, export definitions etc and without the ces nodes can’t do a failover so CES keeps its cluster-wide state in a small Scale file system that every protocol node mounts
./spectrumscale config protocols -f cesSharedRoot -m /ibm/cesSharedRoot
We will now use two floating IP’s known as CES pool so this acts as a load balancer and just the VIP Is exposed out so NFS clients can be served
./spectrumscale config protocols -e 172.16.15.100,172.16.15.101
Set the shared root to our cesSharedroot FS
./spectrumscale config protocols -f cesSharedRoot -m /gpfs/cesSharedRoot
Enable NFS
./spectrumscale config protocols -e 172.16.15.100,172.16.15.101
Check the result
./spectrumscale node list
Install
Let’s now run a precheck and commit the changes in our toolkit to live configuration
./spectrumscale install --precheck
Run the install
./spectrumscale install
Check the result
./spectrumscale node list
To check whether the install landed well we just run
mmgetstate -a
Check stats of cluster
mmclscluster
This shows the CES1 and CES2 are now running in our management network 172.16.11.X however we should be running it on the deamon network 172.16.13.X so lets change it
Stop GPFS on the CES nodes only
mmshutdown -N sitea-ces01.lab.internal,sitea-ces02.lab.internal
Shutdown Cluster
mmshutdown -aMove them to the daemon network using mmchnode
mmchnode --daemon-interface=sitea-ces01-d.lab.internal -N sitea-ces01.lab.internal
mmchnode --daemon-interface=sitea-ces02-d.lab.internal -N sitea-ces02.lab.internal
Start them again
mmstartup -aCheck the cluster status
mmgetstate -a & mmlscluster
The above output shows that we have switched to daemon network 172.16.13.X
Deploy it via toolkit
./spectrumscale deploy --precheck
./spectrumscale deploy 
Verification
Check service status of CES
mmces state show -a
Check the CES nodes
mmces node list
To check ces addresses
mmces address list
Check services on ces
mmces service list -a
CES Failover test
mmces service list -aSuspend CES01
mmces node suspend -N sitea-ces01.lab.internal
Check CES failover
mmces address listThis shows our CES01 is now in a suspended state and CES02 has taken over
CES Resume test
So, let’s bring back the node to CES
mmces node resume -N sitea-ces01.lab.internal
Check CES failover
So this now shows back CES nodes are back
mmces address list
Setting CES coverage
mmces address policy even-coverage
We can also run a rebalance on CES to ensure nodes get equal traffic
mmces address move --rebalance
Run mmlsconfig to see the whole cluster configuration
mmlsconfig 
Run a health check
mmgetstate -a; mmhealth cluster show; mmces address list; mmces state show -a
That’s it about configuring CES nodes and the procedure to failover CES