The Hidden vMotion PortGroup Misconfiguration Behind Packet Bursts on the defaultTcpipStack

vMotion on the default stack is WRONG but then why is vSAN on the default stack during VCF deployment.??

The reason is default stack is the only stack guaranteed to be fully initialised early in the boot process which is necessary for the vSAN disk group bringup and cluster membership . vMotion can be routed (cross‑site, stretched cluster, etc.) so VMware gives it a dedicated TCP/IP stack

vMotion will run for years without anyone questioning how it actually flows across the network. And because vMotion “just works,” odd misconfigurations often go unnoticed until you start seeing huge packet bursts and Packet errors in ESXTOP at times with No CRC errors, no drops, nothing on the physical side. So the usual suspects like bad SFPs, dodgy cables, or HBA issues wont make sense.

The pattern will show up during heavy activity but storage latency which involves vMotion and SDRS movements, that’s when it clicks that these weren’t physical errors at all they were software‑path drops inside the ESXi networking stack.

So when you check the VMkernel interfaces and their TCP/IP stack bindings, the culprit behind is the vMotion Traffic settings..

Correct Setting

On The VCF, installer puts the install this way and this is correct

• vSAN → Default TCP/IP stack
• Management → Default TCP/IP stack
• vMotion → vMotion TCP/IP stack

Lets check its via the command line.

To list the vmkernel interfaces, use the esxcfg-vmknic command with the -l option:

esxcfg-vmknic -l

In this output, the defaultTcpipStack shows there are two vmk adapaters connected to it and this is the correct config when a vSAN network is configured.

root@vvf-esx01:~] esxcfg-vmknic -l

-vmknic -l
Interface  Port Group/DVPort/Opaque Network        IP Family IP Address                              Netmask         Broadcast       MAC Address       MTU     TSO MSS   Enabled Type                NetStack
vmk0       24                                      IPv4      172.16.11.21                            255.255.255.0   172.16.11.255   00:50:ac:10:0b:15 1500    65535     true    STATIC              defaultTcpipStack
vmk2       8                                       IPv4      172.16.13.21                            255.255.255.0   172.16.13.255   00:50:56:6e:0e:aa 9000    65535     true    STATIC              defaultTcpipStack
vmk1       0                                       IPv4      172.16.12.21                            255.255.255.0   172.16.12.255   00:50:56:6a:94:1e 9000    65535     true    STATIC              vmotion
[root@vvf-esx01:~]
[root@vvf-esx01:~]

On the UI, this will look like this.

Wrong Setting

But when we manually config a vMotion vmkernel adapters, this is what happens if you aren’t careful enough

• Management → Default TCP/IP stack
• vMotion → Default TCP/IP stack
• vSAN → Default TCP/IP stack

In this output, the defaultTcpipStack shows there are three vmk adapters connected to it

root@vvf-esx01:~] esxcfg-vmknic -l
Interface  Port Group/DVPort/Opaque Network        IP Family IP Address                              Netmask         Broadcast       MAC Address       MTU     TSO MSS   Enabled Type                NetStack
vmk0       24                                      IPv4      172.16.11.21                            255.255.255.0   172.16.11.255   00:50:ac:10:0b:15 1500    65535     true    STATIC              defaultTcpipStack
vmk2       8                                       IPv4      172.16.13.21                            255.255.255.0   172.16.13.255   00:50:56:6e:0e:aa 9000    65535     true    STATIC              defaultTcpipStack
vmk1       0                                       IPv4      172.16.12.21                            255.255.255.0   172.16.12.255   00:50:56:6a:94:1e 9000    65535     true    STATIC              defaultTcpipStack
[root@vvf-esx01:~]
[root@vvf-esx01:~]

So how did this happen ?

After we create a vSwitch, we usually add a portgoup such as vMotion and just assign to a vlan 1612 butif we dont pick the TCP/IP stack to vMotion w ewill end up vMOtion created on the default tcp IP stack

Running a packet capture on the vmkernel vMotion network

Port 8000 = classic vMotion
Port 902 = encrypted vMotion

Run this while vMotion is active:

pktcap-uw --vmk vmk1 --proto 6 --port 8000 -o /tmp/vmotion-vmk1.pcap

If capture is empty → vMotion may be encrypted

pktcap-uw --vmk vmk1 --proto 6 --port 902 -o /tmp/vmotion-vmk1.pcap

If you see packets on TCP 8000, you are watching the vMotion engine doing its job

To lookup the correct switch port ID, just run this command

[root@vvf-esx01:~] net-stats -l
PortNum          Type SubType SwitchName       MACAddress         ClientName
2214592519          4       0 DvsPortset-0     00:50:56:9f:06:75  vmnic1
2214592521          4       0 DvsPortset-0     00:50:56:9f:f7:8a  vmnic0
67108875            3       0 DvsPortset-0     00:50:ac:10:0b:15  vmk0
67108876            3       0 DvsPortset-0     00:50:56:6a:94:1e  vmk1
67108877            3       0 DvsPortset-0     00:50:56:6e:0e:aa  vmk2
67108882            5       9 DvsPortset-0     00:50:56:aa:08:80  vcf9-ops.eth0

Get the switch port ID and now you can start pktcap-uw.

pktcap-uw --switchport 67108882 --capture VnicRx,VnicTx -o - |  tcpdump-uw -enr -

If you wish to run a query against a mac or ip that can be done here.

pktcap-uw --switchport 67108882 --capture VnicRx,VnicTx -o - |  tcpdump-uw -enr - 'host 172.16.11.10'

Stop pktcap-uw tracing with the kill command:

kill $(lsof |grep pktcap-uw |awk ‘{print $1}’| sort -u)

 To check that all pktcap-uw traces are stopped:

lsof |grep pktcap-uw |awk ‘{print $1}’| sort -u