At this point, we have two Scale sites – Site A and Site B but they dont know about each other. For remote mount, each site will see each other’s filesystem.
If you are coming from the vSAN terminology, this is exactly the vSAN mesh topoology we are dealing here. The ability to mount a remote vSAN datastore is basically what this AFM is all about.. Its fairly easy set to setup a vSAN Mesh but here everywhing behined the covers works with keys being exchanged so lets see what AFM is all about..
Run the mmlscluster to see the nodes in Site A
Run the mmlscluster to see the nodes in Site B
mmlsmount all -L also shows the daemon IPs, the ones each file system is mounted over.
Run mmgetstate -a to check the live stats of nodes
Check the sites can reach each other over the daemon network.
mmremotecluster add sitea.lab.internal -n sitea-mgmt01-d,sitea-nsd01-d,sitea-nsd02-d -k /root/sitea.pub
mmremotefs add sitea_prod_scale_fs -f sitea_prod_scale_fs -C sitea.lab.internal -T /gpfs/sitea_prod_scale_fs -A yes
mmmount sitea_prod_scale_fs -a
df -h | grep gpfs
Step 1 – Generate the keys
To link a remote cluster or link clusters we need a key or a token that will need to exchanged between these sites before it can talk. So, this generated using mmauth
mmauth needs to be run on both sites to generate a key
mmremotecluster add sitea.lab.internal -n sitea-mgmt01-d,sitea-nsd01-d,sitea-nsd02-d -k /root/sitea.pub
mmremotefs add sitea_prod_scale_fs -f sitea_prod_scale_fs -C sitea.lab.internal -T /gpfs/sitea_prod_scale_fs -A yes
mmmount sitea_prod_scale_fs -a
df -h | grep gpfsTo switch on remote access for a local site we issue
mmremotecluster add sitea.lab.internal -n sitea-mgmt01-d,sitea-nsd01-d,sitea-nsd02-d -k /root/sitea.pub
mmremotefs add sitea_prod_scale_fs -f sitea_prod_scale_fs -C sitea.lab.internal -T /gpfs/sitea_prod_scale_fs -A yes
mmmount sitea_prod_scale_fs -a
df -h | grep gpfsThe above commands need to be executed on both sites . The first line creates site’s key and the second switches on remote access
Commit the key on both sites
mmauth genkey commit
Site A
Site B
Now both clusters have this key so we just activvate it by running commit on both sites
mmauth genkey commit
To see the key’s fingerprint (SHA digest)
mmauth show .
Step 2: swap the keys
Both clusters now have a local key and are ready to accept remote clusters so we now need to do a swap of these keys. It lives in /var/mmfs/ssl/id_rsa.pub.
On sitea-mgmt01, send siteA’s key to siteB
mmremotecluster add sitea.lab.internal -n sitea-mgmt01-d,sitea-nsd01-d,sitea-nsd02-d -k /root/sitea.pub
mmremotefs add sitea_prod_scale_fs -f sitea_prod_scale_fs -C sitea.lab.internal -T /gpfs/sitea_prod_scale_fs -A yes
mmmount sitea_prod_scale_fs -a
df -h | grep gpfs
On siteb-mgmt01, send Site B’s key to Site A
scp /var/mmfs/ssl/id_rsa.pub sitea-mgmt01:/root/siteb.pub
Now, both sites have each other’s key
Step3 : Trust the keys
All we now need is to setup a trust between them..
On siteb-mgmt01:
mmauth add sitea.lab.internal -k /root/sitea.pub
This tells siteB that the cluster called sitea.lab.internal is allowed to connect, and this is its key. It’s like adding a trusted certificate to a store.
On sitea-mgmt01, do the same for Site B
mmauth add siteb.lab.internal -k /root/siteb.pub
Step 4 – Mounting the remote file system
The current layout of Site and Site B file systems look like this
mmremotecluster add sitea.lab.internal -n sitea-mgmt01-d,sitea-nsd01-d,sitea-nsd02-d -k /root/sitea.pub
mmremotefs add sitea_prod_scale_fs -f sitea_prod_scale_fs -C sitea.lab.internal -T /gpfs/sitea_prod_scale_fs -A yes
mmmount sitea_prod_scale_fs -a
df -h | grep gpfsmmremotecluster add sitea.lab.internal -n sitea-mgmt01-d,sitea-nsd01-d,sitea-nsd02-d -k /root/sitea.pub
mmremotefs add sitea_prod_scale_fs -f sitea_prod_scale_fs -C sitea.lab.internal -T /gpfs/sitea_prod_scale_fs -A yes
mmmount sitea_prod_scale_fs -a
df -h | grep gpfsSite A mounts Site B’s file system
mmauth grant sitea.lab.internal -f siteb_prod_scale_fs
───── WHO ───── ────── WHAT ──────
- WHO gets access: siteA (the visitor).
- WHAT they get:
siteb_prod_scale_fs(siteB’s own file system).
On siteb-mgmt01:
mmauth grant sitea.lab.internal -f siteb_prod_scale_fs
Tell Site A where Site B is and give it Site B’s key
mmremotecluster add siteb.lab.internal -n siteb-mgmt01-d,siteb-nsd01-d,siteb-nsd02-d -k /root/siteb.pub
Add Site B’s file system
On sitea-mgmt01:
mmremotefs add siteb_prod_scale_fs -f siteb_prod_scale_fs -C siteb.lab.internal -T /gpfs/siteb_prod_scale_fs -A yesMount it by running
mmmount siteb_prod_scale_fs -aIf you’ve set up iSCSI before, it’s the same idea. The storage side allows the client, then the client discovers the target, logs in and mounts.
Step 5: Site B mounts Site A’s file system
This is exactly the same as Step 4, with the names swapped. We need it so either site can take over later.
On sitea-mgmt01:
On siteb-mgmt01:
mmremotecluster add sitea.lab.internal -n sitea-mgmt01-d,sitea-nsd01-d,sitea-nsd02-d -k /root/sitea.pub
mmremotefs add sitea_prod_scale_fs -f sitea_prod_scale_fs -C sitea.lab.internal -T /gpfs/sitea_prod_scale_fs -A yes
mmmount sitea_prod_scale_fs -a
df -h | grep gpfs
Step 6: check it
See which clusters a site trusts and what it shares so run mmauth show all to show it and it siteB now trusts siteA, and siteA has read-write access to siteb_prod_scale_fs.
See which remote clusters a site is linked to:
mmremotecluster show all