Part 17 – Health Checks and Troubleshooting

In the GUI, we can see the Home (dashboard) and Monitoring → Events.

Run Cluster health check


root@sitea-mgmt01:~# mmhealth cluster show

Component           Total         Failed       Degraded        Healthy          Other
-------------------------------------------------------------------------------------
NODE                    5              0              0              2              3
GPFS                    5              0              0              2              3
NETWORK                 5              0              0              5              0
FILESYSTEM              2              0              0              2              0
DISK                   11              0              0             11              0
CES                     2              0              0              0              2
CESIP                   1              0              0              1              0
FILESYSMGR              2              0              0              2              0
GUI                     1              0              1              0              0
PERFMON                 5              0              0              5              0
THRESHOLD               5              0              0              5              0

Check every node, component by component

oot@sitea-mgmt01:~# mmhealth node show -N all

After powering the lab up, the GUI showed DEGRADED with this event

mmhealth event show gui_refresh_task_failed

root@sitea-mgmt01:~# mmhealth event show gui_refresh_task_failed
Event Name:              gui_refresh_task_failed
Description:             One or more GUI refresh tasks failed, which means that data in the GUI is outdated.
Cause:                   There can be several reasons.
User Action:             Check whether there is additional information available by using the '/usr/lpp/mmfs/gui/cli/lstasklog [taskname]' command. Then, run the specified task manually on the CLI by using the '/usr/lpp/mmfs/gui/cli/runtask [taskname] --debug' command. Check the GUI logs under '/var/log/cnlog/mgtsrv' and contact IBM Support if this error persists or occurs more often.
Severity:                WARNING
State:                   DEGRADED

Find the cause in the GPFS log

 tail -30 /var/adm/ras/mmfs.log.latest

Here GPFS started before the GUI web service was listening on port 443 so just re-run a failed task

root@sitea-mgmt01:~# /usr/lpp/mmfs/gui/cli/runtask CLUSTER_CONFIG --debug
debug: locale=en_US
debug: Running 'mmsdrquery 'sdrq_cluster_info' all ' on node localhost
debug: Running 'mmsdrquery 'sdrq_nsd_info' all ' on node localhost
debug: Running 'mmlscluster -Y ' on node localhost
debug: Running 'mmsdrquery 'sdrq_node_info' all ' on node localhost
debug: Running 'mmlsnodeclass 'GUI_MGMT_SERVERS' -Y ' on node localhost
debug: Running 'mmlsnodeclass 'GUI_SERVERS' -Y ' on node localhost
EFSSG1000I The command completed successfully.
root@sitea-mgmt01:~#

IBM Support Data

When you open an IBM support case, they ask for a gpfs.snap, a bundle of logs, configuration and state from the cluster. gpfs -a will run the snap across all nodes

oot@sitea-mgmt01:~# gpfs.snap -a
gpfs.snap: started at Sat Oct 17 08:24:52 PM BST 2026.
Gathering common data...
Gathering Linux specific data...
Gathering extended network data...
Gathering local callhome data...
Gathering local gds data...
Gathering local perfmon data...
Gathering local audit_watch data...
Gathering local sysmon data...
Gathering local cnss data...
Gathering local gui data...
Gathering trace reports and internal dumps...
Gathering Transparent Cloud Tiering data at level BASIC...
gpfs.snap:  The Transparent Cloud Tiering snap software was not detected on node sitea-mgmt01.lab.internal
Gathering QoS data at level FULL...
gpfs.snap:  No QoS configuration was found for this cluster.
gpfs.snap:  QoS configuration collection complete.
Gathering cluster wide gui data...

Quick Health Check

mmgetstate -a                         # every node active
mmhealth cluster show                 # nothing FAILED or DEGRADED
mmlsdisk <filesystem> -e              # no output = all disks up
mmlsmount all -L                      # file systems mounted where expected
mmces state show -a                   # CES/NFS healthy
mmces address list                    # floating IPs spread across CES nodes

Cluster and nodes Check

mmgetstate -a                         # every node active
mmgetstate -s                          # Quorum summary
mmlscluster                           #	Cluster name and ID
mmlsconfig                             #Cluster config settings      

Health monitoring (mmhealth)

mmhealth cluster show                   # health of the whole cluster
mmhealth node show                       # All components on this node
mmhealth node show -N all                # All components on all node
 

GUI Health events

systemctl is-active gpfsgui                 # GUI service on the GUI node
/usr/lpp/mmfs/gui/cli/lstasklog            # Recent GUI refresh tasks and results    

Main Logs

/var/adm/ras/mmfs.log.latest               # found on all nodes 
/var/log/ganesha.log                       # NFS mount problems, slowness, grace   

NFS Issues

showmount -e <ip> from the client           # Is NFS answering?
mmces address list                          # show address
mmces service list -a                        # is nfs running
mmces state show -a                       # is nfs healthy
mmnfs export list                          #NFS exports and allowed clients

Ports 2049 (NFS) and 111 (portmapper) must be open

root@sitea-mgmt01:~# rpcinfo -p sitea-ces01-d

(Visited 4 times, 1 visits today)

By Ash Thomas

Ash Thomas is a seasoned IT professional with extensive experience as a technical expert, complemented by a keen interest in blockchain technology.