Looking at the ESXi VMkernel network path you will notice it consists of Netpoll threads and Tx threads. Netpoll threads receive traffic from an ESXi host perspective where Tx threads transmit data from a VM to another VM or physical component.
By default, each VM is armed with only one Tx thread. As network packets are transmitted from the VM towards the pNIC layer via the VMkernel, ESXi consumes CPU cycles. These cycles, or CPU time, will be accounted for by the VM itself. Tx threads are identified in esxtop in the CPU view as NetWorld-VM-XXX. This ensures that you have a clear picture of what the costs are of transmitting large numbers of network packets from that specific VM. It allows you to have a better understanding of a VM that is constrained by the amount of CPU time that is spent on the transmission of data.
Again, only one Tx thread is spun up by default. That correlates with one CPU core. This is why the NetWorld will not trespass the ±100% of %USED.
In the screenshot above, the VM in question was running the transmit side of the packet-generator test. The NetWorld-VM-69999 world was constantly running up to 100%. This is a clear example of a VM being constrained by only one Tx thread. A relatively quick solution is to add an additional Tx thread. You can add more as needs require. Looking at the network view in esxtop, you will be able to see what vNIC is processing the largest amount of network I/O. In this specific case, we knew exactly what vNIC was in extra need of network processing power.
Additional Tx threads
You can add an additional Tx thread per vNIC. This is configured as an advanced parameter in the VM configuration. The ethernetX.ctxPerDev = 1 advanced setting is used for this. The ‘X’ stands for the vNIC for which the parameter is set. You can configure each vNIC with a separate Tx thread. However, that will create unnecessary Tx threads in your VM and potentially inefficiently consume CPU time, because not every vNIC is likely to require its own Tx thread. It really is a setting that is driven by demand. If your workload running in the VMs has a large appetite for network I/O, take a closer look at what vNIC could benefit from additional Tx threads.
Once the additional Tx thread(s) are configured, you want to verify that it is activated. Additional Tx threads will appear in esxtop in the CPU view as NetWorld-Dev-<id>-Tx. By being added as a separate world, a clear overview can be generated of which NetWorld is processing the majority of network I/O as a result of the CPU usage associated with that thread.
In this screenshot, you will notice that the additional Tx thread is active and processing network I/O. This is one way to determine if your advanced setting is working correctly. You can also use a net-stats command to do so.
More information…
…can be found in the vSphere 6.5 Host Resources Deep Dive book that is available on Amazon!
What is the number (i.e. 71932) on the Networld-VM-71932 process? Or, the Networld-Dev-67108879-Tx? It isn’t the VM ID, I checked that. I don’t know how you correlate these numbers to a specific VM.