Every Scale cluster has a key that is its own so when two clusters need to talk to each other, they exchange these keys to talk. The certificate behind each key is valid for 10 years and will need to be rotated else the remote mounts fail. If it expires, or is replaced carelessly, remote mounts fail the next time a node reboots or remounts.
In this blog, we are going to assume Site A key is nearing expiration, and we are going to rotate the keys.
Site Architecture
Location of the Keys
These keys are located in /var/mmfs/ssl
Procedure at a glance
Each cluster’s key is independent, so you rotate one cluster at a time so here we will rotate siteA
- Check the current key and its expiry on siteA
- Back up /var/mmfs/ssl and keep a copy of the current public key
- Create the new key on siteA (mmauth genkey new). The old and new keys are both accepted now
- Copy the new public key to siteB under a clear name (sitea__.pub)
- Update siteB in both places: mmremotecluster update and mmauth update
- Check that siteB shows the new digest in both places
- Commit on siteA (mmauth genkey commit). Only the new key works from now on
- To rotate siteB, follow the same steps with the roles swapped.
Lets check the expiration date on our certificate
oot@sitea-mgmt01:~#
root@sitea-mgmt01:~# openssl x509 -enddate -noout -in /var/mmfs/ssl/id_rsa_committed.cert
notAfter=Oct 15 10:14:40 2036 GMT
root@sitea-mgmt01:~#This shows our certe expires in 2040, but we are going to rotate it now
Back up the keys
Copy the current key to a location such as /root/keys
root@sitea-mgmt01:# mkdir -p /root/keys
root@sitea-mgmt01:# cp -n /var/mmfs/ssl/id_rsa_committed.pub /root/keys/sitea_2026-10-18_af95288b.pub
root@sitea-mgmt01:#
root@sitea-mgmt01:# ls -l /root/keys/
total 8
-rw-r--r-- 1 root root 1808 Oct 18 12:57 sitea_2026-10-18_af95288b.pub
-rw-r--r-- 1 root root 1808 Oct 18 13:01 siteb_2026-10-18_4d81aa5b.pub
root@sitea-mgmt01:~#Backup the whole folder using zip if required
root@sitea-mgmt01:/var/mmfs/ssl# tar czf /root/keys/sitea_ssl_backup_2026-10-18.tgz -C /var/mmfs ssl
root@sitea-mgmt01:/var/mmfs/ssl# ls -l /root/keys/
total 24
-rw-r--r-- 1 root root 1808 Oct 18 12:57 sitea_2026-10-18_af95288b.pub
-rw-r--r-- 1 root root 16072 Oct 18 13:30 sitea_ssl_backup_2026-10-18.tgz
-rw-r--r-- 1 root root 1808 Oct 18 13:01 siteb_2026-10-18_4d81aa5b.pub
root@sitea-mgmt01:/var/mmfs/ssl#Verification
On sitea-mgmt01 as root, check the remote mounts attachd by running mmlsmount all -L. SiteA’s file system should show mounts from siteB’s nodes as well as siteA’s, and siteb_prod_scale_fs should be mounted on siteA.
Make a note of the current key by running mmauth show .
Note the SHA digest, in site A shows 3db and site B shows b72
root@sitea-mgmt01:~# mmauth show .
Cluster name: sitea.lab.internal (this cluster)
Cipher list: AUTHONLY
SHA digest: af95288ba5300ed97f1341df59895f0ba10c6817e40655e07c944ca11c8893db
File system access: (all rw)root@siteb-mgmt01:~# mmauth show .
Cluster name: siteb.lab.internal (this cluster)
Cipher list: AUTHONLY
SHA digest: 4d81aa5b957f2f44985c465b028b6c79d663f3d67792d3dc577a25c9e3306b72
File system access: (all rw)
root@siteb-mgmt01:~#Run mmauth show and mmremote for Site A to show what each site info holds of each other
Create the new key on siteA
Generate a new key by running mmauth genkey new
root@sitea-mgmt01:~# mmauth genkey new
mmauth: Command successfully completed
root@sitea-mgmt01:~#Run mmauth show . again and siteA now has two keys: the current af95288b… and the new f05e1896…. Both are accepted, so nothing has broken.
root@sitea-mgmt01:~#^C
root@sitea-mgmt01:~# mmauth show .
Cluster name: sitea.lab.internal (this cluster)
Cipher list: AUTHONLY
SHA digest: af95288ba5300ed97f1341df59895f0ba10c6817e40655e07c944ca11c8893db
SHA digest (new): f05e1896f41646fcf9de7f91c864cf0ad5919431cd587823741c1c9c42c221d0
File system access: (all rw)
root@sitea-mgmt01:~#The keys are stored in /var/vmmfs/ssl and the new key we just generated is id_rsa_new.pub.
This is where people break things: running mmauth genkey commit now, before siteB has the new key.
Copy the new key to siteB
drwx------ 2 root root 4096 Oct 18 13:17 stage
root@sitea-mgmt01:/var/mmfs/ssl#
root@sitea-mgmt01:/var/mmfs/ssl# cp -n /var/mmfs/ssl/id_rsa_new.pub /root/keys/sitea_2026-10-18_f05e1896.pub
root@sitea-mgmt01:/var/mmfs/ssl#
root@sitea-mgmt01:/var/mmfs/ssl# scp /root/keys/sitea_2026-10-18_f05e1896.pub root@siteb-mgmt01:/root/keys/
sitea_2026-10-18_f05e1896.pub 100% 1808 3.0MB/s 00:00
root@sitea-mgmt01:/var/mmfs/ssl#
root@sitea-mgmt01:/var/mmfs/ssl#Update siteB in both places
Now we are going to update the keys we got from Site A in into site B.
login to siteb-mgmt01 and run mmauth show all
root@siteb-mgmt01:~/keys#
root@siteb-mgmt01:~/keys# mmauth show all
Cluster name: sitea.lab.internal
Cipher list: AUTHONLY
SHA digest: af95288ba5300ed97f1341df59895f0ba10c6817e40655e07c944ca11c8893db
File system access: siteb_prod_scale_fs (rw, root allowed)
Cluster name: siteb.lab.internal (this cluster)
Cipher list: AUTHONLY
SHA digest: 4d81aa5b957f2f44985c465b028b6c79d663f3d67792d3dc577a25c9e3306b72
File system access: (all rw)
root@siteb-mgmt01:~/keys#
root@siteb-mgmt01:~/keys#siteB uses siteA’s key in two places, so both need updating
When siteB mounts siteA’s file system
mmremotecluster update sitea.lab.internal -k /root/keys/sitea_2026-10-18_f05e1896.pubWhen siteB lets siteA in to siteB’s file system:
mmauth update sitea.lab.internal -k /root/keys/sitea_2026-10-18_f05e1896.pubCheck both digests on siteB
mmremotecluster show sitea.lab.internal
mmauth show sitea.lab.internal SiteB holds siteA’s new key in both places: mmremotecluster and mmauth both show f05e1896…. siteB is ready, so it’s now safe to commit on siteA.
Commit on siteA
Now we are good to commit using mmauth genkey commit
root@sitea-mgmt01:/var/mmfs/ssl# mmauth genkey commit
mmauth: Propagating the cluster configuration data to all affected nodes.
mmauth: Command successfully completed
root@sitea-mgmt01:/var/mmfs/ssl# mmauth show .
Cluster name: sitea.lab.internal (this cluster)
Cipher list: AUTHONLY
SHA digest: f05e1896f41646fcf9de7f91c864cf0ad5919431cd587823741c1c9c42c221d0
File system access: (all rw)
root@sitea-mgmt01:/var/mmfs/ssl#Existing mounts keep running through a commit, because they’re already authenticated so in production we don’t unmount it but just an example is shown showing unmount and mount
oot@siteb-mgmt01:# ls /gpfs/sitea_prod_scale_fs
finance from-siteb.txt nfstest test.txt
root@siteb-mgmt01:#
root@siteb-mgmt01:# mmumount sitea_prod_scale_fs -N siteb-mgmt01.lab.internal
Sun Oct 18 01:58:33 PM BST 2026: mmumount: Unmounting file systems ...
root@siteb-mgmt01:# mmmount sitea_prod_scale_fs -N siteb-mgmt01.lab.internal
Sun Oct 18 01:58:38 PM BST 2026: mmmount: Mounting file systems ...
root@siteb-mgmt01:~# ls /gpfs/sitea_prod_scale_fs
finance from-siteb.txt nfstest test.txtThis is how we rotate scert so now if we have Site B cert expired, we will generate the new key from site B and scp to Site A and the using the same mmremotecluster and mmauth update we will update keys and finally we then commit the key from Site B .. Just one key remains in mmauth output after the commit
In the final blog coming next, we are now to upgrade our Scale cluster