Part 7: Creating the Scale File Systems

We can create the file systems using the spectrum scale toolkit or the manual way. I tried the toolkit but i think i prefer the manual method still to do the scale file system creation

So here we are creating two file systems siteb_prod_scale_fs and cesSharedRoot. The siteb_prod_scale_fs will hold all our folders and files however the cesSharedRoot file system just holds the metadata.

Run mmgetstate -a to check if all nodes are Active

What is done in production is the use of a Stanzafile which is basically a text file containing all the details of each disk and its job.

Run lsblk on nsd01 and nsd02 to check the presentation of luns.

JavaScript

ssh siteb-nsd01 lsblk -o NAME,HCTL
ssh siteb-nsd02 lsblk -o NAME,HCTL

ssh siteb-nsd01 lsblk -o NAME,HCTL
ssh siteb-nsd02 lsblk -o NAME,HCTL

Both nodes match: sdb is LUN 0 and sdl is LUN 10, the same order everywhere.

Create the stanza file for the main file system (8 disks)..In this example nsd=siteb_scale_mgmt01 would mean primary NSD server and siteb-nsd01 would mean backup copy so treat this like fault domain.

JavaScript

root@siteb-mgmt01:~# cat nsd-prod-siteb.txt
%nsd: device=/dev/sdb nsd=siteb_scale_nsd01 servers=siteb-mgmt01,siteb-nsd01 usage=metadataOnly failureGroup=1 pool=system
%nsd: device=/dev/sdc nsd=siteb_scale_nsd02 servers=siteb-mgmt01,siteb-nsd02 usage=metadataOnly failureGroup=2 pool=system
%nsd: device=/dev/sdd nsd=siteb_scale_nsd03 servers=siteb-nsd01,siteb-nsd02 usage=dataOnly failureGroup=1 pool=data
%nsd: device=/dev/sde nsd=siteb_scale_nsd04 servers=siteb-nsd02,siteb-nsd01 usage=dataOnly failureGroup=2 pool=data
%nsd: device=/dev/sdf nsd=siteb_scale_nsd05 servers=siteb-nsd01,siteb-nsd02 usage=dataOnly failureGroup=1 pool=data
%nsd: device=/dev/sdg nsd=siteb_scale_nsd06 servers=siteb-nsd02,siteb-nsd01 usage=dataOnly failureGroup=2 pool=data
%nsd: device=/dev/sdh nsd=siteb_scale_nsd07 servers=siteb-nsd01,siteb-nsd02 usage=dataOnly failureGroup=1 pool=data
%nsd: device=/dev/sdi nsd=siteb_scale_nsd08 servers=siteb-nsd02,siteb-nsd01 usage=dataOnly failureGroup=2 pool=data
root@siteb-mgmt01:~#

root@siteb-mgmt01:~# cat nsd-prod-siteb.txt
%nsd: device=/dev/sdb nsd=siteb_scale_nsd01 servers=siteb-mgmt01,siteb-nsd01 usage=metadataOnly failureGroup=1 pool=system
%nsd: device=/dev/sdc nsd=siteb_scale_nsd02 servers=siteb-mgmt01,siteb-nsd02 usage=metadataOnly failureGroup=2 pool=system
%nsd: device=/dev/sdd nsd=siteb_scale_nsd03 servers=siteb-nsd01,siteb-nsd02 usage=dataOnly failureGroup=1 pool=data
%nsd: device=/dev/sde nsd=siteb_scale_nsd04 servers=siteb-nsd02,siteb-nsd01 usage=dataOnly failureGroup=2 pool=data
%nsd: device=/dev/sdf nsd=siteb_scale_nsd05 servers=siteb-nsd01,siteb-nsd02 usage=dataOnly failureGroup=1 pool=data
%nsd: device=/dev/sdg nsd=siteb_scale_nsd06 servers=siteb-nsd02,siteb-nsd01 usage=dataOnly failureGroup=2 pool=data
%nsd: device=/dev/sdh nsd=siteb_scale_nsd07 servers=siteb-nsd01,siteb-nsd02 usage=dataOnly failureGroup=1 pool=data
%nsd: device=/dev/sdi nsd=siteb_scale_nsd08 servers=siteb-nsd02,siteb-nsd01 usage=dataOnly failureGroup=2 pool=data
root@siteb-mgmt01:~#

Create the stanza file for the CES file system (3 disks). CES holds metadata so we dont need to split them

JavaScript

root@siteb-mgmt01:~# cat nsd-ces-siteb.txt
%nsd: device=/dev/sdj nsd=siteb_scale_nsd09 servers=siteb-nsd01,siteb-nsd02 usage=dataAndMetadata failureGroup=1 pool=system
%nsd: device=/dev/sdk nsd=siteb_scale_nsd10 servers=siteb-nsd02,siteb-nsd01 usage=dataAndMetadata failureGroup=2 pool=system
%nsd: device=/dev/sdl nsd=siteb_scale_nsd11 servers=siteb-nsd01,siteb-nsd02 usage=dataAndMetadata failureGroup=1 pool=system
root@siteb-mgmt01:~#

root@siteb-mgmt01:~# cat nsd-ces-siteb.txt
%nsd: device=/dev/sdj nsd=siteb_scale_nsd09 servers=siteb-nsd01,siteb-nsd02 usage=dataAndMetadata failureGroup=1 pool=system
%nsd: device=/dev/sdk nsd=siteb_scale_nsd10 servers=siteb-nsd02,siteb-nsd01 usage=dataAndMetadata failureGroup=2 pool=system
%nsd: device=/dev/sdl nsd=siteb_scale_nsd11 servers=siteb-nsd01,siteb-nsd02 usage=dataAndMetadata failureGroup=1 pool=system
root@siteb-mgmt01:~#

So we now have 2 files

JavaScript

root@siteb-mgmt01:# ls -l
total 20
-rw-r--r-- 1 root root 109 Oct 13 18:52 nodes-siteb.txt
-rw-r--r-- 1 root root 375 Oct 14 18:54 nsd-ces-siteb.txt
-rw-r--r-- 1 root root 942 Oct 14 18:52 nsd-prod-siteb.txt
drwxr-xr-x 4 root root 4096 Oct 11 12:04 scale
drwx------ 3 root root 4096 Oct 11 11:27 snap
root@siteb-mgmt01:#

root@siteb-mgmt01:# ls -l
total 20
-rw-r--r-- 1 root root 109 Oct 13 18:52 nodes-siteb.txt
-rw-r--r-- 1 root root 375 Oct 14 18:54 nsd-ces-siteb.txt
-rw-r--r-- 1 root root 942 Oct 14 18:52 nsd-prod-siteb.txt
drwxr-xr-x 4 root root 4096 Oct 11 12:04 scale
drwx------ 3 root root 4096 Oct 11 11:27 snap
root@siteb-mgmt01:#

To turn luns into NSD’s, we run mmcrnsd command.-F tells it to read the disks from the stanza file.

JavaScript

mmcrnsd -F /root/nsd-prod-siteb.txt

mmcrnsd -F /root/nsd-prod-siteb.txt

To turn luns into NSD’s, we run mmcrnsd command.-F tells it to read the disks from the stanza file.

To check all nsds we run mmlsnd. This shows the NSD is now ready for assignment into a file system.

JavaScript

mmlsnsd

mmlsnsd

Scale file systems are created using a command called mmcrfs -h

JavaScript

root@siteb-mgmt01:~# mmcrfs -h
Usage:
   mmcrfs Device {"DiskDesc[;DiskDesc...]" | -F StanzaFile}
          [-A {yes | no | automount}] [-B BlockSize] [-D {posix | nfs4}]
          [-E {yes | no}] [-i InodeSize] [-j {cluster | scatter}]
          [-k {posix | nfs4 | all}] [-K {no | whenpossible | always}]
          [-L LogFileSize] [-m DefaultMetadataReplicas]
          [-M MaxMetadataReplicas] [-n NumNodes] [-Q {yes | no}]
          [-p afmAttribute=Value[,afmAttribute=Value...]...]
          [-r DefaultDataReplicas] [-R MaxDataReplicas]
          [-S {yes | no | relatime}] [-T Mountpoint] [-t DriveLetter]
          [-v {yes | no}] [-z {yes | no}] [--auto-inode-limit | --noauto-inode-limit]
          [--filesetdf | --nofilesetdf] [--flush-on-close | --noflush-on-close]
          [--inode-limit MaxNumInodes[:NumInodesToPreallocate]]
          [--log-replicas LogReplicas] [--metadata-block-size BlockSize]
          [--mount-priority Priority] [--nfs4-owner-write-acl {yes | no}]
          [--perfileset-quota | --noperfileset-quota]
          [--version VersionString] [--write-cache-threshold HAWCThreshold]
root@siteb-mgmt01:~#

root@siteb-mgmt01:~# mmcrfs -h
Usage:
   mmcrfs Device {"DiskDesc[;DiskDesc...]" | -F StanzaFile}
          [-A {yes | no | automount}] [-B BlockSize] [-D {posix | nfs4}]
          [-E {yes | no}] [-i InodeSize] [-j {cluster | scatter}]
          [-k {posix | nfs4 | all}] [-K {no | whenpossible | always}]
          [-L LogFileSize] [-m DefaultMetadataReplicas]
          [-M MaxMetadataReplicas] [-n NumNodes] [-Q {yes | no}]
          [-p afmAttribute=Value[,afmAttribute=Value...]...]
          [-r DefaultDataReplicas] [-R MaxDataReplicas]
          [-S {yes | no | relatime}] [-T Mountpoint] [-t DriveLetter]
          [-v {yes | no}] [-z {yes | no}] [--auto-inode-limit | --noauto-inode-limit]
          [--filesetdf | --nofilesetdf] [--flush-on-close | --noflush-on-close]
          [--inode-limit MaxNumInodes[:NumInodesToPreallocate]]
          [--log-replicas LogReplicas] [--metadata-block-size BlockSize]
          [--mount-priority Priority] [--nfs4-owner-write-acl {yes | no}]
          [--perfileset-quota | --noperfileset-quota]
          [--version VersionString] [--write-cache-threshold HAWCThreshold]
root@siteb-mgmt01:~#

Create the main file system siteb_prod_scale_fs

JavaScript

mmcrfs siteb_prod_scale_fs -F /root/nsd-prod-siteb.txt -m 2 -T /gpfs/siteb_prod_scale_fs -A yes

mmcrfs siteb_prod_scale_fs -F /root/nsd-prod-siteb.txt -m 2 -T /gpfs/siteb_prod_scale_fs -A yes

-m 2 keeps 2 copies of metadata.
-T sets where it mounts.
-A yes mounts it automatically at boot.

Create the main file system cesSharedRoot

JavaScript

mmcrfs cesSharedRoot -F /root/nsd-ces-siteb.txt -m 2 -T /gpfs/cesSharedRoot -A yes

mmcrfs cesSharedRoot -F /root/nsd-ces-siteb.txt -m 2 -T /gpfs/cesSharedRoot -A yes

Check the NSDs
oot@siteb-mgmt01:~# mmlsnsd

File system Disk name NSD servers

cesSharedRoot siteb_scale_nsd09 siteb-nsd01.lab.internal,siteb-nsd02.lab.internal
cesSharedRoot siteb_scale_nsd10 siteb-nsd02.lab.internal,siteb-nsd01.lab.internal
cesSharedRoot siteb_scale_nsd11 siteb-nsd01.lab.internal,siteb-nsd02.lab.internal
siteb_prod_scale_fs siteb_scale_nsd01 siteb-mgmt01.lab.internal,siteb-nsd01.lab.internal
siteb_prod_scale_fs siteb_scale_nsd02 siteb-mgmt01.lab.internal,siteb-nsd02.lab.internal
siteb_prod_scale_fs siteb_scale_nsd03 siteb-nsd01.lab.internal,siteb-nsd02.lab.internal
siteb_prod_scale_fs siteb_scale_nsd04 siteb-nsd02.lab.internal,siteb-nsd01.lab.internal
siteb_prod_scale_fs siteb_scale_nsd05 siteb-nsd01.lab.internal,siteb-nsd02.lab.internal
siteb_prod_scale_fs siteb_scale_nsd06 siteb-nsd02.lab.internal,siteb-nsd01.lab.internal
siteb_prod_scale_fs siteb_scale_nsd07 siteb-nsd01.lab.internal,siteb-nsd02.lab.internal
siteb_prod_scale_fs siteb_scale_nsd08 siteb-nsd02.lab.internal,siteb-nsd01.lab.internal

Mount both file systems on all nodes
-a mounts it on all 3 nodes.

mmmount siteb_prod_scale_fs -a

Then the CES one:

mmmount cesSharedRoot -a

Check if they’re mounted

mmlsmount all -L

So, we have now reached the final phase of both site A and site B deployments and ready for the AFM.. Next, in Part 8, we link the two sites, so they trust each other so keep reading.

Tasks	siteA (toolkit)	siteB (stanza files)
NSDs	sitea_scale_nsd01–11	siteb_scale_nsd01–11
Main file system	/gpfs/sitea_prod_scale_fs	/gpfs/siteb_prod_scale_fs
CES file system	/gpfs/cesSharedRoot	/gpfs/cesSharedRoot
Metadata copies	2	2