TCP Segmentation Offload in ESXi explained

TCP Segmentation Offload (TSO) is the equivalent to TCP/IP Offload Engine (TOE) but more modeled to virtual environments, where TOE is the actual NIC vendor hardware enhancement. It is also known as Large Segment Offload (LSO). But what does it do?

When a ESXi host or a VM needs to transmit a large data packet to the network, the packet must be broken down to smaller segments that can pass all the physical switches and possible routers in the network along the way to the packet’s destination. TSO allows a TCP/IP stack to emit larger frames, even up to 64 KB, when the Maximum Transmission Unit (MTU) of the interface is configured for smaller frames. The NIC then divides the large frame into MTU-sized frames and prepends an adjusted copy of the initial TCP/IP headers. This process is referred to as segmentation.

When the NIC supports TSO, it will handle the segmentation instead of the host OS itself. The advantage being that the CPU can present up to 64 KB of data to the NIC in a single transmit-request, resulting in less cycles being burned to segment the network packet using the host CPU. To fully benefit from the performance enhancement, you must enable TSO along the complete data path on an ESXi host. If TSO is supported on the NIC it is enabled by default.

The same goes for TSO in the VMkernel layer and for the VMXNET3 VM adapter but not per se for the TSO configuration within the guest OS. To verify that your pNIC supports TSO and if it is enabled on your ESXi host, use the following command: esxcli network nic tso get. The output will look similar the following screenshot, where TSO is enabled for all available pNICs or vmnics.

In this case, TSO is supported by the NIC and therefore enabled by default. The next step is to verify whether TSO is active within the VMkernel layer. You can check that using the following command: esxcli system settings advanced list -o /Net/UseHwTSO. The output will tell you the current configuration values for TSO within the VMkernel layer.

As expected, the default value is set to ‘1’, meaning that TSO will be active within the VMkernel if it is available on the pNIC.

If you do experience issues that could be related to TSO or situations where TSO would not be beneficial, it is possible to disable it. The easiest way to change is in the VMkernel layer. Since the VMkernel differentiates TSO for the IPv4 and IPv6 stack, you can disable TSO for either one using the following commands for specific pNICs:
esxcli network nic software set –ipv4tso=0 -n vmnicX
esxcli network nic software set –ipv6tso=0 -n vmnicX

Just change the ‘0’ value to ‘1’ to enable it. Another way to disable TSO is to adjust the advanced system setting value /Net/UseHwTSO to ‘0’.

Even though TSO is supported and enabled on the NIC, enabled within the VMkernel and a VMXNET3 virtual adapter is used for the VM, you should also check the guest OS setting. Typically, all common Windows versions and Linux distributions support TSO. To enable or disable TSO in a Linux operating system, use the command ethtool: ethtool -K ethX tso on/off. Where X is the correct interface number with in the Linux OS.

Looking at the latest Windows OS versions, TSO is enabled by default looking at the IPv4 TSO Offload setting. You are even able to enable or disable TSO for both IPv4 and IPv6 in the network and sharing center on the Windows control panel. Just select the properties of the correct network adapter and click configure and browse to the advanced tab.

More information…

…can be found in the vSphere 6.5 Host Resources Deep Dive book that is available on Amazon!