Why IOPS=1 Matters Still

When it comes to optimizing VMware storage performance, the Round Robin Path Selection Policy (PSP) is a key tool in your toolkit. Yet, many admins overlook one simple setting that can dramatically improve I/O performance: setting the IOPS limit to 1.

Round Robin is one of VMware’s built-in multipathing policies. Its job is simple: Distribute I/O operations evenly across all available storage paths.

By default in ESXi the Round Robin IOPS limit is set to 1000 meaning that the path only changes after 1000 I/O operations have occurred. The result is basically the same performance as running a single path in my experience. Adjusting the limit from 1000 to 1 can result in much better performance.

You can check the current IOPS setting and the naa ID we’ll need with this command

esxcli storage nmp device list

esxcli storage nmp device list

In the output you should find a line similar to this one.

naa.6589cfc000000f0033f8cf3c39a68aa7
Device Display Name: FreeNAS iSCSI Disk (naa.6589cfc000000f0033f8cf3c39a68aa7)
Storage Array Type: VMW_SATP_ALUA
Storage Array Type Device Config: {implicit_support=on; explicit_support=off; explicit_allow=on; a lua_followover=on; action_OnRetryErrors=off; {TPG_id=1,TPG_state=AO}}
Path Selection Policy: VMW_PSP_RR
Path Selection Policy Device Config: {policy=iops,iops=1000,bytes=10485760,useANO=0; lastPathIndex=2: NumIOsPending=0,numBytesPending=0}
Path Selection Policy Device Custom Config:
Working Paths: vmhba66:C10:T0:L0, vmhba66:C0:T0:L0, vmhba66:C15:T0:L0, vmhba66:C5:T0:L0
Is USB: false

To verify the round robin settings

#Check RR & IOPS:

esxcli storage nmp device list | grep -A10 naa
esxcli storage nmp psp roundrobin deviceconfig get --device <naa.id>

#For a batch of luns you can run:

for i in $(esxcli storage nmp device list | grep "naa." | awk '{print $3}'); do 
  echo "=== $i ==="; 
  esxcli storage nmp psp roundrobin deviceconfig get --device $i | grep IOOperationLimit; 
done

<Sample output>
=== naa.600xxxxx1 ===
IOOperationLimit: 1
=== naa.600xxxxx2 ===
IOOperationLimit: 1000
=== naa.600xxxxx3 ===
IOOperationLimit: 1
...

#Check RR & IOPS:

esxcli storage nmp device list | grep -A10 naa
esxcli storage nmp psp roundrobin deviceconfig get --device <naa.id>

#For a batch of luns you can run:

for i in $(esxcli storage nmp device list | grep "naa." | awk '{print $3}'); do 
  echo "=== $i ==="; 
  esxcli storage nmp psp roundrobin deviceconfig get --device $i | grep IOOperationLimit; 
done

<Sample output>
=== naa.600xxxxx1 ===
IOOperationLimit: 1
=== naa.600xxxxx2 ===
IOOperationLimit: 1000
=== naa.600xxxxx3 ===
IOOperationLimit: 1
...

To change the IOPS from 1000 to 1 enter this command, replacing XXXX with the naa ID:

Check ALUA paths:

esxcli storage nmp device list | grep -A10 <naa.id>
# Look for Path Selection Policy: VMW_PSP_RR
# Look for "Path State: Active (I/O)" vs "Active Unoptimized"


Monitor during load / boot storm:

esxcli storage core device stats get -d <naa.id>
# QueueFull, NoCommand, ReadLatency, WriteLatency
If latency spikes + QueueFull occur, IOPS=1 helps reduce it dramatically.

Change Default Parameters for Latency Round Robin

So if you have some array such as SVC that need latency round robin settings configured , this setting will do it. No host reboots are required. Note that only iops, bytes, and useANO are tunable.
Everything else (policy, samplingCycles, latencyEvalTime) is calculated automatically by VMware.

https://techdocs.broadcom.com/us/en/vmware-cis/vsphere/vsphere/8-0/vsphere-storage-8-0/understanding-multipathing-and-failover-in-the-esxi-environment/viewing-and-managing-storage-paths-on-esxi-hosts.html

esxcli storage nmp device set --device --psp VMW_PSP_RR \
--psp-option "iops=1" \
--psp-option "bytes=10485760" \
--psp-option "useANO=0"

esxcli storage nmp device set --device --psp VMW_PSP_RR \
--psp-option "iops=1" \
--psp-option "bytes=10485760" \
--psp-option "useANO=0"

This will automatically generate:

• policy=iops
• samplingCycles=16
• latencyEvalTime=180000
• curSamplingCycle values
  (Because they are internally derived by the PSP.)

You do not set these manually — VMware handles them based on the PSP engine.

Sample output
naa.60050768108123456789012345678901
   Device Display Name: IBM Fibre Channel Disk (naa.60050768108123456789012345678901)
   Storage Array Type: VMW_SATP_DEFAULT_AA
   Storage Array Type Device Config: {action_OnRetryErrors=off}
   Path Selection Policy: VMW_PSP_RR
   Path Selection Policy Device Config: {policy=iops,iops=1,bytes=10485760,
       samplingCycles=16,curSamplingCycle=7,latencyEvalTime=180000,useANO=0;
       CurrentPath=vmhba2:C0:T1:L0: NumIOsPending=0,latency=0}
   Working Paths: vmhba1:C0:T1:L0,
                  vmhba2:C0:T1:L0
   Is Local SAS Device: false
   Is Boot USB Device: false
   Device Max Queue Depth: 64
   No of outstanding IOs with competing worlds: 0

For any P/VRDM’s, they are best suited to remain on default IOPS of 1000 or as per Broadcom it should be set to 200 for better failover results thou I’ve always seen RDM’s at 1000.

https://knowledge.broadcom.com/external/article/323117/adjusting-round-robin-iops-limit-from-de.html

There is a Powercli script provided in the kb article that should help you automate the process of changing it.