IBM

Part 14 – Adding New Storage to a Scale File System

By Ash Thomas

We are going to expand our /gpfs/siteb_prod_scale_fs by adding a new disk.

On the SAN side, a new 15 GB LUN has been presented to our NSD servers. I’m using an iSCSI SAN here and skipping the LUN creation itself, since we covered it in the SAN post.

Scan for the new LUN

Rescan each server attached to the SAN.

ssh -t awk@siteb-nsd01 'sudo iscsiadm -m session --rescan && lsblk'
ssh -t awk@siteb-nsd02 'sudo iscsiadm -m session --rescan && lsblk'
ssh -t awk@siteb-nsd02 'sudo iscsiadm -m session --rescan && lsblk'

The new LUN shows up as a 15 GB disk,/dev/sdm on nsd01.

mmlsnd shows how the file system uses its 8 disks,

awk@siteb-mgmt01:~$ mmlsnsd

 File system   Disk name       NSD servers
------------------------------------------------------------------------------
 cesSharedRoot siteb_scale_nsd09 siteb-nsd01.lab.internal,siteb-nsd02.lab.internal
 cesSharedRoot siteb_scale_nsd10 siteb-nsd02.lab.internal,siteb-nsd01.lab.internal
 cesSharedRoot siteb_scale_nsd11 siteb-nsd01.lab.internal,siteb-nsd02.lab.internal
 siteb_prod_scale_fs siteb_scale_nsd01 siteb-mgmt01.lab.internal,siteb-nsd01.lab.internal
 siteb_prod_scale_fs siteb_scale_nsd02 siteb-mgmt01.lab.internal,siteb-nsd02.lab.internal
 siteb_prod_scale_fs siteb_scale_nsd03 siteb-nsd01.lab.internal,siteb-nsd02.lab.internal
 siteb_prod_scale_fs siteb_scale_nsd04 siteb-nsd02.lab.internal,siteb-nsd01.lab.internal
 siteb_prod_scale_fs siteb_scale_nsd05 siteb-nsd01.lab.internal,siteb-nsd02.lab.internal
 siteb_prod_scale_fs siteb_scale_nsd06 siteb-nsd02.lab.internal,siteb-nsd01.lab.internal
 siteb_prod_scale_fs siteb_scale_nsd07 siteb-nsd01.lab.internal,siteb-nsd02.lab.internal
 siteb_prod_scale_fs siteb_scale_nsd08 siteb-nsd02.lab.internal,siteb-nsd01.lab.internal

There are now 6 data disks and 2 system disks. System disk holds the system’s metadata like inode info etc and the real data is in the data disks


awk@siteb-mgmt01:~$
awk@siteb-mgmt01:~$  mmlsdisk siteb_prod_scale_fs
disk         driver   sector     failure holds    holds                            storage
name         type       size       group metadata data  status        availability pool
------------ -------- ------ ----------- -------- ----- ------------- ------------ ------------
siteb_scale_nsd01 nsd         512           1 yes      no    ready         up           system
siteb_scale_nsd02 nsd         512           2 yes      no    ready         up           system
siteb_scale_nsd03 nsd         512           1 no       yes   ready         up           data
siteb_scale_nsd04 nsd         512           2 no       yes   ready         up           data
siteb_scale_nsd05 nsd         512           1 no       yes   ready         up           data
siteb_scale_nsd06 nsd         512           2 no       yes   ready         up           data
siteb_scale_nsd07 nsd         512           1 no       yes   ready         up           data
siteb_scale_nsd08 nsd         512           2 no       yes   ready         up           data

Create a stanza file

We pick the next disk pattern here nsd12 because nsd09-11 went in for our cesShared pool

cat > /root/nsd12.stanza <<'EOF'
%nsd:
  device=/dev/sdm
  nsd=siteb_scale_nsd12
  servers=siteb-nsd01.lab.internal,siteb-nsd02.lab.internal
  usage=dataOnly
  failureGroup=1
  pool=data
EOF

Create the NSD

mmcrnsd along with the Stanza file creates the NSD

root@siteb-mgmt01:~# mmcrnsd -F /root/nsd12.stanza
mmcrnsd: Processing disk sdm
mmcrnsd: Propagating the cluster configuration data to all
  affected nodes.  This is an asynchronous process.
root@siteb-mgmt01:~#

Run mmlsnsd and it shows now a free disk

root@siteb-mgmt01:~# mmlsnsd
cesSharedRoot siteb_scale_nsd09 siteb-nsd01.lab.internal,siteb-nsd02.lab.internal
cesSharedRoot siteb_scale_nsd10 siteb-nsd02.lab.internal,siteb-nsd01.lab.internal
cesSharedRoot siteb_scale_nsd11 siteb-nsd01.lab.internal,siteb-nsd02.lab.internal
siteb_prod_scale_fs siteb_scale_nsd01 siteb-mgmt01.lab.internal,siteb-nsd01.lab.internal
siteb_prod_scale_fs siteb_scale_nsd02 siteb-mgmt01.lab.internal,siteb-nsd02.lab.internal
siteb_prod_scale_fs siteb_scale_nsd03 siteb-nsd01.lab.internal,siteb-nsd02.lab.internal
siteb_prod_scale_fs siteb_scale_nsd04 siteb-nsd02.lab.internal,siteb-nsd01.lab.internal
siteb_prod_scale_fs siteb_scale_nsd05 siteb-nsd01.lab.internal,siteb-nsd02.lab.internal
siteb_prod_scale_fs siteb_scale_nsd06 siteb-nsd02.lab.internal,siteb-nsd01.lab.internal
siteb_prod_scale_fs siteb_scale_nsd07 siteb-nsd01.lab.internal,siteb-nsd02.lab.internal
siteb_prod_scale_fs siteb_scale_nsd08 siteb-nsd02.lab.internal,siteb-nsd01.lab.internal
(free disk) siteb_scale_nsd12 siteb-nsd01.lab.internal,siteb-nsd02.lab.internal

Next, we add it to the file system using command mmadddisk

root@siteb-mgmt01:~# mmadddisk siteb_prod_scale_fs -F /root/nsd12.stanza

The following disks of siteb_prod_scale_fs will be formatted on node siteb-mgmt01:
    siteb_scale_nsd12: size 15360 MB
Extending Allocation Map
Checking Allocation Map for storage pool data
  35 % complete on Sat Oct 17 22:01:36 2026
  73 % complete on Sat Oct 17 22:01:41 2026
  99 % complete on Sat Oct 17 22:01:46 2026
 100 % complete on Sat Oct 17 22:01:46 2026
Completed adding disks to file system siteb_prod_scale_fs.
mmadddisk: Propagating the cluster configuration data to all
  affected nodes.  This is an asynchronous process.
root@siteb-mgmt01:~#

NSD created and we can see nsd12 is now assigned

root@siteb-mgmt01:~# mmlsnsd

 File system   Disk name       NSD servers
------------------------------------------------------------------------------
 cesSharedRoot siteb_scale_nsd09 siteb-nsd01.lab.internal,siteb-nsd02.lab.internal
 cesSharedRoot siteb_scale_nsd10 siteb-nsd02.lab.internal,siteb-nsd01.lab.internal
 cesSharedRoot siteb_scale_nsd11 siteb-nsd01.lab.internal,siteb-nsd02.lab.internal
 siteb_prod_scale_fs siteb_scale_nsd01 siteb-mgmt01.lab.internal,siteb-nsd01.lab.internal
 siteb_prod_scale_fs siteb_scale_nsd02 siteb-mgmt01.lab.internal,siteb-nsd02.lab.internal
 siteb_prod_scale_fs siteb_scale_nsd03 siteb-nsd01.lab.internal,siteb-nsd02.lab.internal
 siteb_prod_scale_fs siteb_scale_nsd04 siteb-nsd02.lab.internal,siteb-nsd01.lab.internal
 siteb_prod_scale_fs siteb_scale_nsd05 siteb-nsd01.lab.internal,siteb-nsd02.lab.internal
 siteb_prod_scale_fs siteb_scale_nsd06 siteb-nsd02.lab.internal,siteb-nsd01.lab.internal
 siteb_prod_scale_fs siteb_scale_nsd07 siteb-nsd01.lab.internal,siteb-nsd02.lab.internal
 siteb_prod_scale_fs siteb_scale_nsd08 siteb-nsd02.lab.internal,siteb-nsd01.lab.internal
 siteb_prod_scale_fs siteb_scale_nsd12 siteb-nsd01.lab.internal,siteb-nsd02.lab.internal

Check file system by running mmlspool

root@siteb-mgmt01:~# mmlspool siteb_prod_scale_fs
Storage pools in file system at '/gpfs/siteb_prod_scale_fs':
Name                    Id   BlkSize Data Meta Total Data in (KB)   Free Data in (KB)   Total Meta in (KB)    Free Meta in (KB)
system                   0      4 MB   no  yes              0              0 (  0%)       20971520       17305600 ( 83%)
data                 65537      4 MB  yes   no      110100480      109584384 (100%)              0              0 (  0%)

root@siteb-mgmt01:~# mmdf siteb_prod_scale_fs
disk                disk size  failure holds    holds           free in KB          free in KB
name                    in KB    group metadata data        in full blocks        in fragments
--------------- ------------- -------- -------- ----- -------------------- -------------------
Disks in storage pool: system (Maximum disk size allowed is 106.74 GB)
siteb_scale_nsd01      10485760        1 yes      no          8642560 ( 82%)         10840 ( 0%)
siteb_scale_nsd02      10485760        2 yes      no          8642560 ( 82%)         10840 ( 0%)
                -------------                         -------------------- -------------------
(pool total)         20971520                              17285120 ( 82%)         21680 ( 0%)

Disks in storage pool: data (Maximum disk size allowed is 221.74 GB)
siteb_scale_nsd12      15728640        1 no       yes        15654912 (100%)          8056 ( 0%)
siteb_scale_nsd03      15728640        1 no       yes        15654912 (100%)          8056 ( 0%)
siteb_scale_nsd07      15728640        1 no       yes        15654912 (100%)          8056 ( 0%)
siteb_scale_nsd05      15728640        1 no       yes        15654912 (100%)          8056 ( 0%)
siteb_scale_nsd06      15728640        2 no       yes        15654912 (100%)          8056 ( 0%)
siteb_scale_nsd04      15728640        2 no       yes        15654912 (100%)          8056 ( 0%)
siteb_scale_nsd08      15728640        2 no       yes        15654912 (100%)          8056 ( 0%)
                -------------                         -------------------- -------------------

The data pool grew by +15 GiB

Scale doesn’t move existing data onto a new disk by itself, but every new write is striped across all disks, the new one included, so usage evens out over time. Existing data movement causes massive performance issues so only run rebalance if the pool is actually full else, we leave it for system to do the data placement.

(Visited 6 times, 1 visits today)

By Ash Thomas

Ash Thomas is a seasoned IT professional with extensive experience as a technical expert, complemented by a keen interest in blockchain technology.

vExpert community knowledge — protected from AI training, bots, and automated harvesting. | BlogArise by Themeansar.