IoT – Everything and everyone smartly connected

The subject Internet of Things (IoT) is intriguing to me, both from a technical and personal point-of-view. IoT to me is where all current cloud initiatives and network virtualization innovations come together, complemented with new technologies. Today, we are realizing highly scalable cloud platforms that are secure and easy to manage as they serve our workloads and store data. Building these scalable cloud platforms and seeing traction on cloud native applications is another push to support IoT innovations. It is fun to see that we are moving towards adopting cloud technologies, but are already on the verge of entering new technology rapids.

Innovation is moving at a scarily fast pace. – Bill Gates

Looking at IoT and its applications, one will quickly notice that the possibilities are immense! The future will be data driven. All our devices, be it a mobile phone, smartwatch, car, plane or refrigerator, will gather data that will lead to more insights on how we as human beings operate and consume. That will lead to services being specifically tailored per individual. And what about enterprises and industrial components? Imagine what can be done in every industry vertical. We can only try to understand the opportunities.

Data in Edge

We may be somewhat dismissive about every device being connected to the ‘internet’, but IoT is already happening. Think about ordering sushi or fast food in selected restaurants where the order is entered using a tablet or similar device. Or what about a hospital where patient data is gathered by sensors and is accessible on mobile devices for the nurses. Think about a car that is collecting data about the driver and his driving characteristics, or the data that is generated by the vehicle itself. Each sensor in a device will collect data, we are talking about a lot of data mining here… Think Gigabytes of data from a single car drive or Terabytes per plane per flight.

The future includes Brontobytes and Geobytes – Michael Dell

That raises the question; is all this data consumable? How are we going to collect all that data? How do we process, analyze and store that data? At first looks, you may think that all the data is ingested in a central cloud environment. But that is not efficient. Because we are talking about a prodigious amount of data it only seems logical to process, analyse and even compress the data locally before sending the important catches of data over to a centralized platform.

Local data processing will require local compute and storage resources, also referred to as edge computing. So even though we have cloud computing as a tool execute data processing, it seems inevitable that data will use local compute and storage power on edge devices. It will be interesting to track the application and development of efficient edge compute solutions (like nano-servers, further usage of Field-programmable Gate Array (FPGA) chipsets, etc.) in the near future.

Moving to edge computing is interesting because today we are in the process of centralizing IT in cloud solutions while IoT innovations will lead to decentral systems, although accompanied by central systems. They will supplement each other.

NFV

A very important factor for IoT and edge devices will be the telecom providers and as they will provide 5G services. The rollout of 5G is a key driver for IoT as it allows for a very low and consistent latency while increasing the bandwidth which is required to support connectivity to the edge devices. This will be a necessity as we will be transferring a lot of data from a massive amount of IoT devices to a centralised cloud platform. The ability to create highly scalable and performant 5G platforms will depend strongly on Network Functions Virtualization (NFV).

Telco operators are working hard on moving from a monolithic telco workload landscape to fully virtualized platforms. VMware helps to thrive NFV adoption with vCloud for NFV that includes vSphere, NSX and related components. It is very important to squeeze every bit out of hardware and the ESXi  / VM layer to realize consolidation ratios and lower costs while building a consistent performant fundament for NFV. That was one of the key reasons for us (Frank and myself) to write the Host Deep Dive book.

In the process of working towards NFV and IoT, are we finally forced to adopt IPv6? I would say ‘yes’! IPv4 will simply not cut it anymore, as we all said well before 2010.

To conclude…

We have enough challenges to deal with, but we can already see new solutions being developed to deal with security and manageability of IoT and edge components. Cloud (native) platforms are gaining traction and so is NFV.

I am still trying to wrap my head around the endless possibilities and try to understand the impact of the IoT and edge wave, powered by powerful NFV and cloud platforms. Very exciting times ahead!! We are only scratching the surface of what can be done with IT.

 

Read More

VXLAN and Multiple Receive Threads explained

The (Dynamic) Netqueue feature in ESXi, which is enabled by default if the physical NIC (pNIC) supports it, allows incoming network packets to be distributed over different queues. Each queue gets its own ESXi thread for packet processing. One ESXi thread represents a CPU core.

However, (Dynamic) NetQueue and VXLAN are not the best of friends when it comes to distributing network I/O over multiple queues. That is because of the way Virtual Tunnel End Points (VTEP) are set up. Within a VMware NSX implementation, each ESXi host in the cluster contains at least one VTEP, dependent upon the NIC load balancing mode chosen. The VTEP is the component that provides the encapsulation and decapsulation for the VXLAN packets. That means all VXLAN network traffic from a VM perspective will traverse the VTEP and the receiving VTEP on another ESXi host.

Therein lies the problem when it comes to NetQueue and the ability to distribute network I/O streams over multiple queues. This is because a VTEP will always have the same MAC address and the VTEP network will have a fixed VLAN tag. MAC address and VLAN tag are the filters most commonly supported by pNICs with VMDq and NetQueue enabled. It will seriously restrict the ability to have multiple queues and thereby will possibly restrict the network performance for your VXLAN networks. VMware NSX now supports multi-VTEPs per ESXi host. This helps slightly as a result of the extra MAC addresses, because of the increased number of VTEPs per ESXi host. NetQueue can therefore have more combinations to filter on. Still, it is far from perfect when it comes to the
desired network I/O parallelism handling using multiple queues and CPU cores. To overcome that challenge, there are some pNICs that support the distributing of queues by filtering on inner (encapsulated) MAC addresses. RSS can do that for you.

(more…)

Read More

Using CPU limits in vCloud for NFV

Looking at the VMware vCloud for NFV proposition, you will notice that vCloud Director (vCD) is one of the options for the Virtualized Infrastructure Manager (VIM) layer based on the ETSI framework.

VMware vCloud NFV supports two integrated virtualized infrastructure managers (VIMs): native VMware vCloud Director or VMware Integrated OpenStack, a full OpenStack implementation that is completely tested and integrated. Both VIMs support templated service descriptions as well as multi-tenancy and robust networking, enabling the automation of on-boarding VNFs with the acceleration of configuring and allocating compute, storage and networking resources.

As mentioned, vCD is used for multi tenancy and providing a management layer for the tenants to spin up new workloads within their own set of resources. Now how the compute resources are provided to the workloads is very interesting in some telco workload cases. vCD provides three different ways to deliver compute resources to the tenant vApps:

  • Allocation Pool
    A percentage of the resources you allocate are committed to the Organization virtual DataCenter (OvDC). You can specify the percentage, which allows you to overcommit resources.
  • Pay-As-You-Go
    Allocated resources are only committed when users create vApps in the OvDC. You can specify the maximum amount of CPU and memory resources to commit to the OvDC.
  • Reservation Pool
    All of the resources you allocate are committed to the OvDC.

More information on allocation models in vCD can be found on Duncan’s blogpost, its one of his older posts but still accurate. The Pay-As-You-Go allocation model seems to be a popular choice because it enforces the entitlement for specific resources per Virtual Machine (VM). It does so by setting a reservation and a limit on each VM in the vApp / OvDC using a configurable vCPU speed. That means a VM can only consume CPU cycles as configured in the OvDC. See the following example to get a better feeling on what is configurable within a Pay-As-You-Go OvDC.

 

Now, bring the fact that limits and reservations are placed on a VM back to the fact that network I/O and the required CPU time to process that network I/O is accounted to the VM. Check the Virtual Machine Tx threads explained post to get a better understanding of how the VMkernel behaves when VMs transmit network packets. Because the CPU cycles used by the Tx threads are accounted to the VM, you should be very careful with applying CPU limits to the VM. CPU limits can seriously impact the network I/O performance and packet rate capability!

A severe performance impact can be expected when you deploy NFV Telco workloads that are known for high network utilization when using CPU limits. A clear example is a virtual Evolved Packet Core (vEPC) node, for instance the Packet Gateway (PGW). VMs used in these nodes are known to have a large appetite for network I/O and may be using Intel DPDK to thrive network packet rates.

Several vCPUs in the VMs will be configured in DPDK to poll a queue. Those vCPUs will claim all the cycles they can get their hands on! You should combine that with the additional CPU time required to process the transmitted network I/Os to fully understand the behaviour of such a VM and its need for CPU time. Only then can you make the correct decisions related to allocation models.

So be aware of the possible implications that CPU limits may introduce on network performance. Maybe it is better for certain telco NFV workloads to opt for another allocation model.

More information on how the VMware vCloud for NFV proposition is helping the telco industry and the LTE/4G and 5G innovations is found here: https://vmware.regalixdigital.com/nfv-ebook/automation.html. Be sure to check it out!

Read More

VMworld 2017 session picks

VMworld is upon us. The schedule builder went live and boy am I excited about VMworld 2017!

SER1872BU

This year Frank and I are presenting the successor of last years session at both VMworlds. We are listed in the schedule builder as SER1872BU – vSphere 6.5 Host Resources Deep Dive: Part 2. We are planning to bring even more ESXi epicness with a slight touch of vSAN and networking information that allows you to prep ESXi to run NFV workloads that drive IoT innovations. Last year we were lucky to have packed rooms in Vegas and Barcelona.

The enthusiasm about our book witnessed so far shows us there is still a lot of love out there for the ESXi hypervisor and ‘under-the-hood’ tech! We are working hard on having an awesome session ready for you!

VMworld Session Picks

This year I want to learn more about NFV, IoT and Edge as I find innovation in these areas intriguing. I found some sessions that look to be very interesting. I supplemented these with talks held by industry titans about various topics. If my schedule lets me, I want to see the following sessions:

  • Leading the 5G and IoT Revolution Through NFV [FUT3215BUby Constantine Polychronopoulos
  • vSAN at the Edge: HCI for Distributed Applications Spanning Retail to IoT [STO2839GU] by Kristopher Groh
  • VMware Cloud Foundation Futures [PBO2797BU] by Raj Yavatkar
  • Machine Learning and Deep Learning on VMware vSphere: GPUs Are Invading the Software-Defined Data Center [VIRT1997BU] by Uday Kurkure and Ziv Kalmanovich
  • Managing Your Hybrid Cloud with VMware Cloud on AWS [LHC2971BU] by Frank Denneman and Emad Younis
  • The Top 10 Things to Know About vSAN [STO1264BU] by Duncan Epping and Cormac Hogan

There are way more interesting sessions going on! Be sure to find and schedule your favorite ones as rooms tend to fill up quickly!

See you in Vegas and Barcelona!!

 

Read More

Virtual Machine Tx threads explained

Looking at the ESXi VMkernel network path you will notice it consists of Netpoll threads and Tx threads. Netpoll threads receive traffic from an ESXi host perspective where Tx threads transmit data from a VM to another VM or physical component.

By default, each VM is armed with only one Tx thread. As network packets are transmitted from the VM towards the pNIC layer via the VMkernel, ESXi consumes CPU cycles. These cycles, or CPU time, will be accounted to the VM itself. Tx threads are identified in esxtop in the CPU view as NetWorld-VM-XXX. This ensure that you to have a clear picture on what the costs are of transmitting large numbers of networks packets from that specific VM. It allows you to have a better understanding if a VM is constrained by the amount of CPU time that is spent on transmission of data.

Again, only one Tx thread is spun up by default. That correlates with one CPU core. This is why the NetWorld will not trespass the ±100% of %USED.

In the screenshot above, the VM in question was running the transmit side of the packet-generator test. The NetWorld-VM-69999 world was constantly running up to 100%. This is a clear example of a VM being constrained by only one Tx thread. A relatively quick solution is to add an additional Tx thread. You can add more as needs require. Looking at the network view in esxtop, you will be able to see what vNIC is processing the largest amount of network I/O. In this specific case, we knew exactly what vNIC was in extra need of network processing power.

Additional Tx threads

You can add an additional Tx thread per vNIC. This is configured as an advanced parameter in the VM configuration. The ethernetX.ctxPerDev = 1 advanced setting is used for this. The ‘X’ stands for the vNIC for which the parameter is set. You can configure each vNIC with a separate Tx thread. However, that will create unnecessary Tx threads in your VM and potentially consume CPU time in an inefficient way, because not every vNIC is likely to require its own Tx thread. It really is a setting that is driven by demand. If your workload running in the VMs has a large appetite for network I/O, take a closer look at what vNIC could benefit from additional Tx threads.

Once the additional Tx thread(s) are configured, you want to verify that it is activated. Additional Tx threads will appear in esxtop in the CPU view as NetWorld-Dev-<id>-Tx. By being added as a separate world, a clear overview can be generated on which NetWorld is processing the majority of network I/O as a result of the CPU usage associated with that thread.

In this screenshot, you will notice that the additional Tx thread is active and processing network I/O. This is one way to determine if your advanced setting is working correctly. You can also use a net-stats command to do so.

More information…

…can be found in the vSphere 6.5 Host Resources Deep Dive book that is available on Amazon!

Read More

Host Resources Deep Dive Released!

The time has finally come that we published our (e)book. It is available via Amazon using the following links:

Paperback Kindle (ebook)
Amazon US Amazon US 
Amazon DE Amazon DE
Amazon Mexico Amazon NL
Amazon UK Amazon UK
 Amazon Japan Amazon Japan
 Amazon India Amazon India

 

Countless hours, weeks, months has gone into this project. It all began in the beginning of 2016. Frank and I had a lot of discussions about consistent performance and how to optimize while keeping consolidation ratios in mind. At the time, I was working on a NFV platform that was plotted on a virtual datacenter and Frank was working on his epic NUMA content. It all led to the idea of writing down our findings in a book.

It is almost like going against the current; we were looking into ESXi host behaviour, while the world is advancing to higher services running on top of vSphere. Why even bother about the hypervisor? It is all commodity after all, right? Well… not per se. It is all about understanding what you are doing on each level, not about just throwing hardware at a performance/capacity requirement. We began to talk about our ideas to our peers to see if they were interesting and feasible to include in a book.

We quickly came to the conclusion that there are a lot of unknowns when it comes to tuning your virtual environment to accompany distributed services like vSAN and NSX, latency sensitive workloads and right-sizing your VMs in general. We hope our book will help vSphere administrators, architects, consultants, aspiring VCDX-es and people eager to learn more about the elements that control the behavior of CPU, memory, storage and network resources.

I am extremely grateful that I was part of realising this book. It is really inspiring to work closely with Frank. He has a tremendous way of expressing his train of thought and has a lot of experience in creating tech books. Our discussions about tech or general topics are always a blast.

All our effort led to our (and my first) publishing which contains:

  • 122.543 words
  • 5217 paragraphs
  • 23 chapters
  • 569 pages
  • 311 screenshots and diagrams

The main challenge was to dig deep in the world of host resources while working a very busy day job at my beloved customers. The effort required was immense. Think a VCDX path times 10. At least, that is how it felt to me. But it was so much fun. I remember a few weeks ago; We were working on the last storage content until deep into the night, like we did almost every day for the last few months, and we were still so psyched at 3.30AM. So even after a very intense period, we still got the kicks out of talking about the content and book in the middle of the night!

We tried to keep the price as low as possible to allow everybody interested in it to be able to buy it. We feel we managed to do that even though the book contains 569 pages. The digital book will following after some PTO and both VMworlds.

As discussed in a previous post; We attended several VMUGs to talk about content included in the book. We are now also confirmed to have a ‘part deux’ of last years VMworld top 10 session this year in Las Vegas and in Barcelona (session 1872). We really look forward seeing you there! Please let us know your thoughts on our book, even write a review on Amazon if you will. It is very rewarding to hear it helped you in some way. Thanks!

Read More