A Tale from the Trench – E1000/VMXnet3

powercli

A silent killer

Was it a disaster situation? A 0-day? or a data center fire? Nope, not a flashy problem or even a spectacular firestorm.

It was a simple change

An inadvertent change of a Microsoft Server 2012 R2 VMware template’s NIC from VMXNET3 to E1000. Effectively reducing the NIC bandwidth from 10Gb to 1Gb as well as other benefits of using the VMXNET3 driver over the emulated driver of the E10oo. This went unnoticed and affected ~50 VM’s.

Slowness

A coworker was troubleshooting backup slowness issues and brought his findings to me.

I used the below PowerCLI command on each vCenter to view and exported to CSV any VM whose NIC was not equal to “Vmxnet3”. Thanks to @LucD22 from a 2012 post https://communities.vmware.com/thread/395849

Connect-VIServer vserver.yourdomain.com
Get-VM | Get-NetworkAdapter | Where-object {$_.Type -ne "Vmxnet3"} | Select @{N="VM";E={$_.Parent.Name}},Name,Type | export-Csv  c:\temp\Network_Interface.csv -NoTypeInformation

Results

The resulting CSV file looks like the below.  Some VM’s needed to stay on e1000 as they were Linux appliances, but approximately 50 Windows VM’s needed to be changed.
niccsv

Resolution

The resolution didn’t turn out to be as easy as just changing the Network Interface to VMXnet3. A PowerCLI one liner would have finished the job in a couple of minutes, but it was not to be.

Changing the VMs existing Network Interface from E1000 to VMXnet3 was not the problem. The VM Operating System was having the issue, 2012R2 didn’t recognize the NIC as having changed. Even after a reboot and scanning for new hardware changes in Device Manager. Since I was going to have to reconfigure the static IP addressing information anyway, I ended up just deleting the existing NIC’s and adding a new one. Booting the VM and re-configuring the IP information. I had maintenance down time at night so the ~50 VM’s were changed starting with higher priority servers. If I had more VM’s to correct I would have looked for a fully automated solution.

Lessons Learned

Templates changes should be reported on and should be checked periodically for adherence to corporate standards as well as industry best practices.

Resources

VMXnet3 vs E1000e  – http://rickardnobel.se/vmxnet3-vs-e1000e-and-e1000-part-1/

What’s faster  – http://www.ntpro.nl/blog/archives/1478-Whats-faster-E1000-or-VMXNET3-lets-see-what-PassMarks-PerformanceTest-says.html

 

Thanks,

Scott Bollinger / kfalconspb / www.bollingerusa.com

 

 

 

 

 

Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s