In this part we upgrade siteA from 5.1.8.2 to 5.2.0.1 with the installation toolkit, online (one node at a time), while NFS clients and siteB’s remote mount stay connected. The upgrade itself is one command is the most interesting bit in this but since it’s got a lot of moving parts, there is lot of reading to do as well.
References
- Upgrading IBM Storage Scale components with the installation toolkit (5.2.0)
- Performing online upgrade by using the installation toolkit (5.1.9)
- Performing offline upgrade or excluding nodes (5.1.2)
- IBM Storage Scale 5.2.0 Concepts, Planning, and Installation Guide (PDF)
- IBM Storage Scale 5.2.0 Administration Guide (PDF)
- IBM Storage Scale 5.2.0.1 Fix Readme
- IBM Storage Scale Software Version Recommendation
- Potential deadlock during and after a level update (5.2.0.0 or newer)
- IBM Spectrum Scale supported upgrade paths (5.1.3)
- IBM Storage Scale FAQ (PDF)
IBM Storage Scale supported upgrade paths
As per the Supported Upgrade paths, this is a simple hop upgrade from 5.1.8X to 5.2.0X
| Upgrading from | To 5.1.8.x | To 5.1.9.x | To 5.2.0.x |
|---|---|---|---|
| 5.1.8.x | ✓ | ✓ | ✓ |
Check the version of Scale
=== mmdiag: version ===
Current GPFS build: "5.1.8.2 ".
Built on Aug 31 2023 at 11:03:05
Running 6 hours 54 minutes 25 secs, pid 3282
root@sitea-mgmt01:~#
root@sitea-mgmt01:~# mmdsh -N all "dpkg-query -W gpfs.base"
sitea-ces01.lab.internal: gpfs.base 5.1.8-2
sitea-ces02.lab.internal: gpfs.base 5.1.8-2
sitea-nsd01.lab.internal: gpfs.base 5.1.8-2
sitea-mgmt01.lab.internal: gpfs.base 5.1.8-2
sitea-nsd02.lab.internal: gpfs.base 5.1.8-2Run a cluster helath check
mmgetstate -a
mmhealth cluster show
mmces state show -a
mmlscluster
root@sitea-mgmt01:~# mmgetstate -a
Node number Node name GPFS state
------------------------------------------
1 sitea-mgmt01-d active
2 sitea-nsd01-d active
3 sitea-nsd02-d active
4 sitea-ces01-d active
5 sitea-ces02-d active
root@sitea-mgmt01:~# mmhealth cluster show
Component Total Failed Degraded Healthy Other
-------------------------------------------------------------------------------------
NODE 5 0 1 2 2
GPFS 5 0 0 2 3
NETWORK 5 0 0 5 0
FILESYSTEM 3 0 0 3 0
DISK 11 0 0 11 0
CES 2 1 0 0 1
CESIP 1 0 0 1 0
FILESYSMGR 2 0 0 2 0
GUI 1 0 1 0 0
PERFMON 5 0 0 5 0
THRESHOLD 5 0 0 5 0
root@sitea-mgmt01:~# mmces state show -a
NODE AUTH BLOCK NETWORK HDFS_NAMENODE AUTH_OBJ NFS OBJ SMB CES
-------------------------- ---------- ---------- --------- --------------- ---------- --------- ---------- ------ -----
sitea-ces01.lab.internal DISABLED DISABLED HEALTHY DISABLED DISABLED HEALTHY DISABLED TIPS TIPS
sitea-ces02.lab.internal DISABLED DISABLED HEALTHY DISABLED DISABLED HEALTHY DISABLED TIPS TIPS
root@sitea-mgmt01:~# mmlscluster
GPFS cluster information
========================
GPFS cluster name: sitea.lab.internal
GPFS cluster id: 6413056013524421807
GPFS UID domain: sitea.lab.internal
Remote shell command: /usr/bin/ssh
Remote file copy command: /usr/bin/scp
Repository type: CCR
Node Daemon node name IP address Admin node name Designation
-----------------------------------------------------------------------------------------
1 sitea-mgmt01-d.lab.internal 172.16.13.11 sitea-mgmt01.lab.internal quorum-manager-perfmon
2 sitea-nsd01-d.lab.internal 172.16.13.13 sitea-nsd01.lab.internal quorum-perfmon
3 sitea-nsd02-d.lab.internal 172.16.13.14 sitea-nsd02.lab.internal quorum-perfmon
4 sitea-ces01-d.lab.internal 172.16.13.15 sitea-ces01.lab.internal perfmon
5 sitea-ces02-d.lab.internal 172.16.13.16 sitea-ces02.lab.internal perfmon
oot@sitea-mgmt01:~# mmlsconfig
Configuration data for cluster sitea.lab.internal:
--------------------------------------------------
clusterName sitea.lab.internal
clusterId 6413056013524421807
autoload yes
profile gpfsProtocolDefaults
dmapiFileHandleSize 32
minReleaseLevel 5.1.8.0
tscCmdAllowRemoteConnections no
ccrEnabled yes
sdrNotifyAuthEnabled yes
maxblocksize 16M
[cesNodes]
maxMBpS 5000
numaMemoryInterleave yes
enforceFilesetQuotaOnRoot yes
workerThreads 512
[common]
tscCmdPortRange 60000-61000
cipherList AUTHONLY
cesSharedRoot /gpfs/cesSharedRoot
cesCidrPool <IP=172.16.15.100><ATTRIBUTE=><GROUP=><PREFIX=>+<IP=172.16.15.101><ATTRIBUTE=><GROUP=><PREFIX=>
cifsBypassTraversalChecking yes
syncSambaMetadataOps yes
cifsBypassShareLocksOnRename yes
adminMode central
File systems in cluster sitea.lab.internal:
-------------------------------------------
/dev/cesSharedRoot
/dev/sitea_prod_scale_fs
root@sitea-mgmt01:~#Checking the portability layer and this show kernel running on 5.15.0-198 . Every node needs a portability layer (the GPFS kernel module) built for its exact kernel. The toolkit rebuilds it during the upgrade, but it can only build on a kernel Scale supports.
root@sitea-mgmt01:/usr/lpp/mmfs/5.2.0.1/ansible-toolkit# mmdsh -N all "uname -r"
sitea-ces01.lab.internal: 5.15.0-198-generic
sitea-ces02.lab.internal: 5.15.0-198-generic
sitea-nsd01.lab.internal: 5.15.0-198-generic
sitea-mgmt01.lab.internal: 5.15.0-198-generic
sitea-nsd02.lab.internal: 5.15.0-198-generic
root@sitea-mgmt01:/usr/lpp/mmfs/5.2.0.1/ansible-toolkit#As per the support matrix, table doesn’t say 5.15.0-198-generic is supported but the latest tested kernel for Ubuntu 22.04.5 is 5.15.0-179-generic so ours is newer so it should be fine, but not a guarantee for 5.2.0.1 . On an untested newer kernel, the risk is that mmbuildgpl fails to build or builds but misbehaves. For production, we certainly will keep nodes on a tested kernel and switched off auto upgrades, so it remains locked.
Download Storage_Scale_Developer-5.2.0 to a path and run the installer and it will extract product packages to /usr/lpp/mmfs/5.2.0.1
root@sitea-mgmt01:~/scale/5.2.0.1# ls -al
total 1861112
drwxr-xr-x 3 root root 4096 Oct 18 18:15 .
drwxr-xr-x 4 root root 4096 Oct 11 12:04 ..
drwx---rwx 2 awk awk 4096 Oct 17 22:40 Storage_Scale_Developer-5.2.0.1-x86_64-Linux
-rwxr-xr-x 1 root root 953117386 Jun 7 2024 Storage_Scale_Developer-5.2.0.1-x86_64-Linux-install
-rw-r--r-- 1 root root 87 Jun 7 2024 Storage_Scale_Developer-5.2.0.1-x86_64-Linux-install.md5
-rw-r--r-- 1 root root 4029 Jun 7 2024 Storage_Scale_Developer-5.2.0.1-x86_64-Linux.README
-rw-rw-r-- 1 awk awk 952626834 Oct 18 18:14 Storage_Scale_Developer-5.2.0.1-x86_64-Linux.zip
-rw-r--r-- 1 root root 1623 Jun 5 2024 Storage_Scale_public_key.pgp
root@sitea-mgmt01:~/scale/5.2.0.1#
root@sitea-mgmt01:~/scale/5.2.0.1#
root@sitea-mgmt01:~/scale/5.2.0.1# ./Storage_Scale_Developer-5.2.0.1-x86_64-Linux-install --silent
Extracting License Acceptance Process Tool to /usr/lpp/mmfs/5.2.0.1 ...
tail -n +678 ./Storage_Scale_Developer-5.2.0.1-x86_64-Linux-install | tar -C /usr/lpp/mmfs/5.2.0.1 -xvz --exclude=installer --exclude=*_rpms --exclude=*_debs --exclude=*rpm --exclude=*gz --exclude=*deb --exclude=*tools* 1> /dev/null
Next is the new toolkit (
Browse to the ansible toolkit
root@sitea-mgmt01:~/scale/5.2.0.1# cd /usr/lpp/mmfs/5.2.0.1/ansible-toolkit
root@sitea-mgmt01:/usr/lpp/mmfs/5.2.0.1/ansible-toolkit# ls -l
total 144Setup the mgmt as setup node
oot@sitea-mgmt01:/usr/lpp/mmfs/5.2.0.1/ansible-toolkit# ./spectrumscale setup -s 172.16.11.11
[ INFO ] Installing prerequisites for install node
[ INFO ] Found existing Ansible installation on system.
[ INFO ] Install Toolkit setup type is set to IBM Storage Scale (default). If an ESS is in the cluster, run this command to set ESS mode: ./spectrumscale setup -s server_ip -st ess
[ INFO ] Your ansible controller node has been configured to use the IP 172.16.11.11 to communicate with other nodes.
[ INFO ] Port 10080 will be used for package distribution.
[ INFO ] SUCCESS
[ INFO ] Tip : Designate protocol, nsd and admin nodes in your environment to use during install:./spectrumscale -v node add <node> -p -a -n
root@sitea-mgmt01:/usr/lpp/mmfs/5.2.0.1/ansible-toolkit#The toolkit detected ansible already now we will use -N to populate the cluster info
./spectrumscale config populate -N sitea-mgmt01.lab.internal
Run ./spectrumscale node list to see if it populates
- Sitea-mgmt01 Admin, Quorum, Manager, NSD, GUI, Perf Mon
- sitea-nsd01 Quorum, NSD, (Perf Mon)
- sitea-nsd02 Quorum, NSD
- sitea-ces01 Protocol
- sitea-ces02 Protocol
- Protocols NFS Enabled
- Export IPs 172.16.15.100, 172.16.15.101
Precheck
Run precheck via./spectrumscale upgrade precheck and this picks an error on rpcbind service running on mgmt01, nsd01 and nsd02 and this is needed for NFS to work. We will just ignore the above alert as it’s a lab and continue the install via ./spectrumscale upgrade run
Rolling Upgrade of GPFS Cluster
This upgrade script works through the nodes one by one: CES IPs moved, GPFS stopped, 5.2.0.1 packages installed, portability layer rebuilt, GPFS started and this will take some time. . If the 198 kernel is a problem, it fails here, and the toolkit stops before touching the other nodes.so the mmbuildgpl / portability layer step on the first node is the first sign of failure if any
root@sitea-mgmt01:/usr/lpp/mmfs/5.2.0.1/ansible-toolkit# ./spectrumscale upgrade run
The upgrade process is divided into several phases. If the nodes being upgraded include both protocol (CES) and non-protocol nodes, then protocol nodes and non-protocol nodes are upgraded in separate phases. If the nodes being upgraded are all of the same type, either protocol or non-protocol, then only the upgrade phases applicable to that node type will be performed. The upgrade process may cause a brief outage of Object, SMB, NFS, HDFS and Performance Monitoring components. Do you really want to begin upgrading? [y/N]: Y
[ INFO ] Logging to file: /usr/lpp/mmfs/5.2.0.1/ansible-toolkit/logs/UPGRADE-18-10-2026_18:54:28.log
[ INFO ]
[ INFO ]
[ INFO ] ***************************
[ INFO ] * Package version check *
[ INFO ] ***************************
[ INFO ]
[ INFO ] Component: GPFS
[ INFO ] ---------------
[ INFO ] Version of gpfs.base installed on sitea-mgmt01.lab.internal: 5.1.8-2. Upgrade needed to 5.2.0-1.
[ INFO ] Version of gpfs.base installed on sitea-nsd01.lab.internal: 5.1.8-2. Upgrade needed to 5.2.0-1.
[ INFO ] Version of gpfs.base installed on sitea-nsd02.lab.internal: 5.1.8-2. Upgrade needed to 5.2.0-1.
[ INFO ] Version of gpfs.base installed on sitea-ces01.lab.internal: 5.1.8-2. Upgrade needed to 5.2.0-1.
[ INFO ] Version of gpfs.base installed on sitea-ces02.lab.internal: 5.1.8-2. Upgrade needed to 5.2.0-1.
[ INFO ] Version of gpfs.gpl installed on sitea-mgmt01.lab.internal: 5.1.8-2. Upgrade needed to 5.2.0-1.
[ INFO ] Version of gpfs.gpl installed on sitea-nsd01.lab.internal: 5.1.8-2. Upgrade needed to 5.2.0-1.
[ INFO ] Version of gpfs.gpl installed on sitea-nsd02.lab.internal: 5.1.8-2. Upgrade needed to 5.2.0-1.
[ INFO ] Version of gpfs.gpl installed on sitea-ces01.lab.internal: 5.1.8-2. Upgrade needed to 5.2.0-1.
[ INFO ] Version of gpfs.gpl installed on sitea-ces02.lab.internal: 5.1.8-2. Upgrade needed to 5.2.0-1.
[ INFO ] Version of gpfs.msg.en-us installed on sitea-mgmt01.lab.internal: 5.1.8-2. Upgrade needed to 5.2.0-1.
[ INFO ] Version of gpfs.msg.en-us installed on sitea-nsd01.lab.internal: 5.1.8-2. Upgrade needed to 5.2.0-1.
[ INFO ] Version of gpfs.msg.en-us installed on sitea-nsd02.lab.internal: 5.1.8-2. Upgrade needed to 5.2.0-1.
[ INFO ] Version of gpfs.msg.en-us installed on sitea-ces01.lab.internal: 5.1.8-2. Upgrade needed to 5.2.0-1.
[ INFO ] Version of gpfs.msg.en-us installed on sitea-ces02.lab.internal: 5.1.8-2. Upgrade needed to 5.2.0-1.
[ INFO ] Version of gpfs.docs installed on sitea-mgmt01.lab.internal: 5.1.8-2. Upgrade needed to 5.2.0-1.Monitor the status of upgrade
Watch the status via UI
After around 20 mins, Scale has been upgraded to 5.2.X
On the UI, it will still take some time before the monitoring perf collectors picks up the changes, so if necessary, run the fix procedure.
The cluster is now fully live on 5.2.X
Post Health Checks
oot@sitea-mgmt01:/usr/lpp/mmfs/5.2.0.1/ansible-toolkit# mmdsh -N all "dpkg-query -W gpfs.base"
sitea-ces02.lab.internal: gpfs.base 5.2.0-1
sitea-ces01.lab.internal: gpfs.base 5.2.0-1
sitea-nsd01.lab.internal: gpfs.base 5.2.0-1
sitea-mgmt01.lab.internal: gpfs.base 5.2.0-1
sitea-nsd02.lab.internal: gpfs.base 5.2.0-1
root@sitea-mgmt01:/usr/lpp/mmfs/5.2.0.1/ansible-toolkit#
#
root@sitea-mgmt01:/usr/lpp/mmfs/5.2.0.1/ansible-toolkit# mmdiag --version
=== mmdiag: version ===
Current GPFS build: "5.2.0.1 ".
Built on May 30 2024 at 16:06:39
Running 51 minutes 7 secs, pid 340601
root@sitea-mgmt01:/usr/lpp/mmfs/5.2.0.1/ansible-toolkit#
oot@sitea-mgmt01:/usr/lpp/mmfs/5.2.0.1/ansible-toolkit# mmgetstate -a
Node number Node name GPFS state
------------------------------------------
1 sitea-mgmt01-d active
2 sitea-nsd01-d active
3 sitea-nsd02-d active
4 sitea-ces01-d active
5 sitea-ces02-d active
root@sitea-mgmt01:/usr/lpp/mmfs/5.2.0.1/ansible-toolkit# mmdsh -N all "dpkg-query -W gpfs.base"
sitea-ces02.lab.internal: gpfs.base 5.2.0-1
sitea-ces01.lab.internal: gpfs.base 5.2.0-1
sitea-nsd01.lab.internal: gpfs.base 5.2.0-1
sitea-mgmt01.lab.internal: gpfs.base 5.2.0-1
sitea-nsd02.lab.internal: gpfs.base 5.2.0-1
root@sitea-mgmt01:/usr/lpp/mmfs/5.2.0.1/ansible-toolkit# mmdiag --version
=== mmdiag: version ===
Current GPFS build: "5.2.0.1 ".
Built on May 30 2024 at 16:06:39
Running 51 minutes 7 secs, pid 340601
root@sitea-mgmt01:/usr/lpp/mmfs/5.2.0.1/ansible-toolkit# mmgetstate -a
Node number Node name GPFS state
------------------------------------------
1 sitea-mgmt01-d active
2 sitea-nsd01-d active
3 sitea-nsd02-d active
4 sitea-ces01-d active
5 sitea-ces02-d active
root@sitea-mgmt01:/usr/lpp/mmfs/5.2.0.1/ansible-toolkit# mmhealth cluster show
Component Total Failed Degraded Healthy Other
----------------------------------------------------------------------------------------------------------------
NODE 5 0 0 0 5
GPFS 5 0 0 0 5
NETWORK 5 0 0 5 0
FILESYSTEM 3 0 0 3 0
DISK 11 0 0 11 0
CES 2 0 0 0 2
CESCLUSTER 1 0 0 1 0
FILESYSMGR 1 0 0 1 0
GUI 1 0 1 0 0
PERFMON 5 0 0 5 0
THRESHOLD 5 0 0 5 0
root@sitea-mgmt01:/usr/lpp/mmfs/5.2.0.1/ansible-toolkit# mmces state show -a
NODE AUTH BLOCK NETWORK HDFS_NAMENODE AUTH_OBJ NFS OBJ S3 SMB CES
-------------------------- ---------- ---------- --------- --------------- ---------- --------- ---------- ---------- ------ -----
sitea-ces01.lab.internal DISABLED DISABLED HEALTHY DISABLED DISABLED HEALTHY DISABLED DISABLED TIPS TIPS
sitea-ces02.lab.internal DISABLED DISABLED HEALTHY DISABLED DISABLED HEALTHY DISABLED DISABLED TIPS TIPS
root@sitea-mgmt01:/usr/lpp/mmfs/5.2.0.1/ansible-toolkit# mces address list
Command 'mces' not found, did you mean:
command 'mcs' from snap maven-central-search (0.11.1)
command 'mccs' from deb mccs (1:1.1-9)
command 'mcs' from deb mono-mcs (6.8.0.105+dfsg-3.2)
command 'mes' from deb mes (0.23-4)
command 'nces' from deb nco (5.0.6-1)
See 'snap info <snapname>' for additional versions.
root@sitea-mgmt01:/usr/lpp/mmfs/5.2.0.1/ansible-toolkit# mmces address list
Address Node Ces Group Attributes
--------------- -------------------------- ----------- ------------
172.16.15.100 sitea-ces01.lab.internal none none
172.16.15.101 sitea-ces02.lab.internal none none
otal 0
root@sitea-mgmt01:/mnt# df -H
Filesystem Size Used Avail Use% Mounted on
tmpfs 823M 1.6M 821M 1% /run
/dev/mapper/ubuntu--vg-ubuntu--lv 40G 18G 21G 47% /
tmpfs 4.2G 29k 4.2G 1% /dev/shm
tmpfs 5.3M 0 5.3M 0% /run/lock
/dev/sda2 2.1G 138M 1.8G 8% /boot
tmpfs 823M 4.1k 823M 1% /run/user/1000
cesSharedRoot 6.5G 3.0G 3.5G 47% /gpfs/cesSharedRoot
sitea_prod_scale_fs 97G 453M 97G 1% /gpfs/sitea_prod_scale_fs
siteb_prod_scale_fs 113G 529M 113G 1% /gpfs/siteb_prod_scale_fs
root@sitea-mgmt01:/mnt# cd /gpfs/sitea_prod_scale_fs/
root@sitea-mgmt01:/gpfs/sitea_prod_scale_fs# ls
finance from-siteb.txt nfstest test.txt
root@sitea-mgmt01:/gpfs/sitea_prod_scale_fs# mmlsconfig minReleaseLevel
minReleaseLevel 5.1.8.0
root@sitea-mgmt01:/gpfs/sitea_prod_scale_fs# mmlsfs sitea_prod_scale_fs -V
flag value description
------------------- ------------------------ -----------------------------------
-V 31.00 (5.1.7.0) File system version
root@sitea-mgmt01:/gpfs/sitea_prod_scale_fs#
Final Commit
Once the upgrade is complete, the final procedure will be to commit the code using command mmchconfig release=LATEST which can be done at a later date. There is no rollback after this step.
mmchconfig: Propagating the cluster configuration data to all
affected nodes. This is an asynchronous process.
root@sitea-mgmt01:/gpfs/sitea_prod_scale_fs#After the file system is migrated, remote nodes that are running earlier GPFS versions will not be able to mount the file system anymore so only do this step after Site B is moved
root@sitea-mgmt01:/gpfs/sitea_prod_scale_fs# mmchfs sitea_prod_scale_fs -V full
mmchfs: Attention: One or more remote clusters are granted access to the file system.
After the file system is migrated, remote nodes that are running earlier
GPFS versions will not be able to mount the file system any more.
File system version 34.00 is not supported on the following 5 remote nodes mounting the file system:
172.16.14.21 siteb-mgmt01-d siteb.lab.internal 32.02
172.16.14.26 siteb-ces02-d siteb.lab.internal 32.02
172.16.14.25 siteb-ces01-d siteb.lab.internal 32.02
172.16.14.23 siteb-nsd01-d siteb.lab.internal 32.02
172.16.14.24 siteb-nsd02-d siteb.lab.internal 32.02
mmchfs: Command failed. Examine previous error messages to determine cause.
root@sitea-mgmt01:/gpfs/sitea_prod_scale_fs#
root@sitea-mgmt01:/gpfs/sitea_prod_scale_fs#Likewise, we will do the same for cesSharedRoot as well.
(Visited 4 times, 1 visits today)