Virtual Machine Tx threads explained

Looking at the ESXi VMkernel network path you will notice it consists of Netpoll threads and Tx threads. Netpoll threads receive traffic from an ESXi host perspective where Tx threads transmit data from a VM to another VM or physical component.

By default, each VM is armed with only one Tx thread. As network packets are transmitted from the VM towards the pNIC layer via the VMkernel, ESXi consumes CPU cycles. These cycles, or CPU time, will be accounted to the VM itself. Tx threads are identified in esxtop in the CPU view as NetWorld-VM-XXX. This ensure that you to have a clear picture on what the costs are of transmitting large numbers of networks packets from that specific VM. It allows you to have a better understanding if a VM is constrained by the amount of CPU time that is spent on transmission of data.

Again, only one Tx thread is spun up by default. That correlates with one CPU core. This is why the NetWorld will not trespass the ±100% of %USED.

In the screenshot above, the VM in question was running the transmit side of the packet-generator test. The NetWorld-VM-69999 world was constantly running up to 100%. This is a clear example of a VM being constrained by only one Tx thread. A relatively quick solution is to add an additional Tx thread. You can add more as needs require. Looking at the network view in esxtop, you will be able to see what vNIC is processing the largest amount of network I/O. In this specific case, we knew exactly what vNIC was in extra need of network processing power.

Additional Tx threads

You can add an additional Tx thread per vNIC. This is configured as an advanced parameter in the VM configuration. The ethernetX.ctxPerDev = 1 advanced setting is used for this. The ‘X’ stands for the vNIC for which the parameter is set. You can configure each vNIC with a separate Tx thread. However, that will create unnecessary Tx threads in your VM and potentially consume CPU time in an inefficient way, because not every vNIC is likely to require its own Tx thread. It really is a setting that is driven by demand. If your workload running in the VMs has a large appetite for network I/O, take a closer look at what vNIC could benefit from additional Tx threads.

Once the additional Tx thread(s) are configured, you want to verify that it is activated. Additional Tx threads will appear in esxtop in the CPU view as NetWorld-Dev-<id>-Tx. By being added as a separate world, a clear overview can be generated on which NetWorld is processing the majority of network I/O as a result of the CPU usage associated with that thread.

In this screenshot, you will notice that the additional Tx thread is active and processing network I/O. This is one way to determine if your advanced setting is working correctly. You can also use a net-stats command to do so.

More information…

…can be found in the vSphere 6.5 Host Resources Deep Dive book that is available on Amazon!

Read More

VMUGs and VMworlds

While being extremely busy to complete our upcoming book, Frank and I are planning to deliver our vSphere 6.5 : Host Resources Deep Dive sessions at several VMUG’s. Just last week we had a blast presenting some Compute and Networking content at one of the largest UserCon VMUGs in the world, the NLVMUG. It was one of the best VMUG UserCons I have ever visited. I do love these days as you really get to connect with your peers and share knowledge and experiences! You can’t afford to miss your local VMUG.

It was very good to see that our room was packed! We got loads of positive feedback which is always nice. It is pretty rewarding to see that the VMware community is taking great interest in our content which is basically about the VMware vSphere layer and how host resources are consumed in ESXi 6.5. To give you some impressions of our session last week, check the pictures below:



After thoroughly enjoying the NLVMUG day, we were on the lookout to attend more VMUGs. So, after having contact with several VMUG leaders, it looks like we are presenting at the following VMUGs:

  • Belgium VMUG in Mechelen, 12th of May  (Didn’t work out planning-wise)
  • German VMUG in Frankfurt, 14th of June
  • UK VMUG in London, 22nd of June

If everything follows through, it will be a very good way to meet up with even more people. Be sure to mark these dates your agenda’s, the listed VMUGs are going to be epic!

VMworld 2017

In other news, we also submitted a session for both VMworld US and EMEA 2017. It will be a sequel to our VMworld 2016 top 10 session. Keep an eye out for session ID 1872vSphere 6.5 Host Resources Deep Dive : Part 2.

We will get the book done long before VMworld, so we are really excited to see the reactions on it. We are working effortlessly, as are our reviewers, to get create a book that can help everybody working with virtual datacenters in their daily job!

Hope to see you at a VMUG or at a VMworld conference!

Read More

AWS direct connect – Connectivity matters!

I had some discussion about AWS (Amazon Web Services) and how to connect to their services, especifically when you run production workloads on virtual machines in AWS. Bringing workloads to public clouds, means that your business and/or your customers are more depended on their (internet) connectivity to be able to reach the workloads running on public cloud environment.

Connectivity matters

There are a multiple solutions out there to make your internet facing connections highly available. Bandwidth-wise there aren’t really any challenges, aside from the costs… in the Netherlands at least. It is easy to get a 1GbE or better connection from your datacenter or office location(s).

The thing we were discussing about, is the latency between you and your public cloud services. Even though it’s strongly depending on what workloads you are planning to run in AWS, you want a decent user experience. Thus a lowest possible network latency towards that workload. That brings us to www.cloudping.info. A nifty web tool to give you an idea on what your latency is to the regions from where AWS offers their services. It’s output looks like this:

awscloudping

Since I’m in the Netherlands, the EU Frankfurt site in Germany is the closest AWS site for me. So an average ping time of 23ms… Note: This number is depending strongly on how your internet provider or your datacenter is connected to AWS via peering on various Internet Exchanges or via transits.

(more…)

Read More

Lab test: vSphere Fault Tolerance performance impact

Triggered by some feedback on the VMware reddit channel, I was wondering what is holding us back in adopting the vSphere Fault Tolerance (FT) feature. Comments on Reddit stated that although the increased availability is desirable, the performance impact is holding them back to actually use it in production environments.

Use cases for FT could be, according to the vSphere 6 documentation center:

  • Applications that need to be available at all times, especially those that have long-lasting client connections that users want to maintain during hardware failure.
  • Custom applications that have no other way of doing clustering or other forms of application resiliency.
  • Cases where high availability might be provided through custom clustering solutions, which are too complicated to configure and maintain.

However, the stated use cases only focus on availability and do not seem to incorporate a performance impact when enabling FT. Is there a sweet-spot for applications that do need high resiliency, but do not require immense performance and could coop with a latency impact due to FT? It really depends on the application workload. A SQL server typically generates more FT traffic then for instance a webserver that primarily transmits. So the impact of enabling FT will impact some workloads more then other.

Requirements

Since the introduction of vSphere 6: Multi-Processor Fault Tolerance (SMP-FT), the requirements for FT are a bit more flexible. The compute maximums for a FT enabled VM are 4 vCPUs and 64GB memory. The use of eager zero thick disks is no longer a requirement. So thin, lazy zeroed thick and eager zero thick provisioned disks are all supported in SMP-FT!
(more…)

Read More

Our VMworld 2016 session (#8430)

Oh how do I look forward to VMworld 2016!! Once more I’m able to visit both VMworld US in Las Vegas and VMworld EMEA in Barcelona. This year it’s not all about listening and peering though as I am presenting together with the one and only Frank Denneman!

Next to the fact that presenting at VMworld will be an awesome experience, it’ll also be good to already convey some thoughts & content from our upcoming book in our VMworld session!!

vmworldvegas

Our VMworld session

It is difficult to get a VMworld session accepted. If I remember correctly, there were 1550+ abstracts submitted and only 611 made it into the catalog.  Setting up a good abstract is critical, so I figured it might help to share our submitted abstract. Still, I’m pretty sure even very good abstracts/sessions were rejected which is a shame. I sincerely hope we still get to see this content at your local VMUG or in a vBrownBag TechTalk at VMworld US or EMEA.

Our submitted abstract:
(more…)

Read More

Jumbo frames and the risks involved

Even though the jumbo frame and the possible gain and risk trade-offs discussion is not new, we found ourselves discussing it yet again. Because we had different opinions, it seems like a good idea to elaborate on this topic.

Let’s have a quick recap on what jumbo frames actually are. Your default MTU (Maximum Transmission Unit) for a ethernet frame is 1500. A MTU of 9000 is referred to as a jumbo frame.

Jumbo frames or 9000-byte payload frames have the potential to reduce overheads and CPU cycles.

Typically, jumbo frames are considered for IP storage networks or vMotion networks. A lot of performance benchmarking is already described on the web. It is funny to see a variety of opinions whether to adopt jumbo frames or not. Check this blogpost and this blogpost on jumbo frames performance compared to a standard MTU size. The discussion if ‘jumbo frames provide a significant performance advantage’ is still up in the air.

There are other techniques to improve network throughput and lower CPU utilization next to jumbo frames. A modern NIC will support the Large Segment Offload (LSO) and Large Receive Offload (LRO) offloading mechanisms. Note: LSO is also referenced as TSO (TCP Segmentation Offload). Both are configurable. LSO/TSO is enabled by default if the used NIC hardware supports it. LRO is enabled by default when using VMXNET virtual machine adapters.

Risks?

Let’s put the performance aspects aside, and let us look into the possible risks involved when implementing jumbo frames. The thing is, in order to be effective, jumbo frames must be enabled end to end in the network path. The main risk when adopting jumbo frames, is that if one component in the network path is not properly configured for jumbo frames, a MTU mismatch occurs.
(more…)

Read More

Stretched cluster with NSX

Last NLVMUG I was talking about stretched clusters. My presentation elaborated somewhat on how VMware NSX can help you deal with challenges that arise when deploying a stretched cluster solution. In this blogpost I want to have a closer look at this specific topic.

A quick understanding about what a stretched cluster solution actually is; it is a vSphere cluster configured in one vCenter instance containing an equal number of hosts from both sites. This allows for disaster avoidance (vMotion) and disaster recovery (vSphere HA) between two geographical separated sites. From the backend infrastructure perspective, your (synchronous replicated) storage and network solutions must span both sites.

Looking into network designs used for stretched clusters, you will typically face challenges like:

  • How do you design for VM mobility over 2 sites, requiring Layer-2 networks between the 2 sites?
  • Stretched Layer-2 networks (VLANs) introduce a higher risk of failure (think Layer-2 loops).
  • How to properly segment applications and/or tentants (customers/business units)?
  • Netwerk flows. What about your egress and ingress connections?

Let’s begin with how a VMware NSX install-base could look like if it is deployed within stretched cluster infrastructure.

Stretched cluster with NSX architecture

A stretched cluster with VMware NSX could look like the following logical overview.
(more…)

Read More

Stretched cluster VM & datastore affinity

When using a vSphere stretched cluster solution, it is important to have your VM(s) and its VMDK(s) affinity aligned in the same datacenter. So if the storage controller in datacenter 1 serves the read/write copy of the datastore, you would like the VM to run on a vSphere host in the same datacenter. This will avoid the storage read IO’s to traverse the inter-datacenter connections, resulting in an obvious impact on performance. With the VM – datastore affinity in place, you will also mitigate the risk of potential VM outage if a datacenter partition (aka split-brain scenario) will occur.

Let me show you what I mean by using a simple logical overview of a stretched cluster infrastructure. The following example is based on an uniform storage backend. More information on uniform and non-uniform metro storage solutions is read here.

What you don’t want:

VM affinity

What you do want:

VM affinity

 

It is perfectly possible to automate the alignment upon… VM creation for example. Needless to say, you will require DRS to run. Preferably in fully automated mode.

(more…)

Read More

vSphere and NFV tuning considerations

In my current role, I am involved in a lot of discussions around network functions virtualization, a.k.a. NFV. Talking about NFV in this post, I mean telco applications. By that I mean applications specifically designed for and used by Communications Service Providers (CSPs) as core applications to, for instance, enable your (mobile) phone to actually be able to call another phone. 🙂

NFV with regards to telco applications is not that mainstream so it seems. The old school native way, having telco specific hardware running line cards, payload servers, etc., obviously is not sustainable looking at the current way we like to do ICT.  On the other hand, it looks like telco application vendors are still finding their way on how to properly adopt virtualization as a technology. So it looks like the level of virtualization adoption for network functions is a few years back in comparison to IT application server virtualization.

But development is rapid, and so it is for NFV. There already is a NFV Architecture Framework created by ETSI. ETSI was selected in November 2012 to be the home of the Industry Specification Group for NFV. The framework is a high-level functional architecture and design philosophy for virtualized network functions and the underlying virtualization infrastructure as shown in the following diagram:

ETSI-NFV

Although there are words that NFV is mostly deployed using a KVM hypervisor working closely with OpenStack as the API framework for NFV,  VMware is looking to hook into the needs of the communications service providers to properly ‘do’ NFV using VMware solutions. Hence the vCloud for NFV suite.

VMware vCloud NFV is a Network Functions Virtualization (NFV) services delivery, operations and management platform, developed for Communications Service Providers (CSPs) who want to reduce infrastructure CapEx and OpEx costs, improve operational agility and monetize new services with rapid time to market.

vcloudnfv

Let’s have a closer look at tuning considerations for vSphere to properly run NFV workloads!
(more…)

Read More

Containers, VMs and unikernels

Last week I had an interesting discussion with a colleague on containers (Docker mostly), VMs, as well as a more recent development in this space called unikernels. Regular geek speak. I’ve mashed up the most interesting parts of the discussion, together with some background information.

 

Containerization

Containerization is lightweight OS virtualization that groups and isolates certain processes and resources from the host operating system and other containers. Containers share the operating system kernel and may share binaries and libraries.

The following image depicts the difference between VMs and containers.
VMs versus containers

(more…)

Read More