Consistent Network Configuration across ESX hosts

One of the key settings from our VMware health check that needed adjusted was the network setup. The goal is to make all connections redundant and constitant across the configuration so I am going to post the recommended config for our particular ESX hosts which are Dell M710's. They have the capability to have 2 sets of Mezzanine cards, however we only have 1 set filled. Of those we are only using the onboard ethernet or A fabric and the ethernet mezzanine or B fabric.

Therefore per the healthcheck tool, here are the recommendations.

----------------

Minimize differences in the network configuration across all hosts in a cluster. Consistent networking configuration across all hosts in a cluster easies administration and troubleshooting. Also, since services like VMotion require portgroups to be named consistently in order for VMotion to work, it is important to have a consistent nnnnnzz

Also, use a consistent naming convention for virtual switches, portgroups, and uplink groups

Product/Version: vSphere 4

VMware vSphere 4 introduces VMware vNetwork Distributed Switches (vDS) and Cisco Nexus 1000V distributed switches which reduce administration time and ensure consistency across the virtual datacenter. Changes to the distributed virtual portgroup are consistently and automatically applied to all hosts that are connected to the distributed switch. Check the licensing requirements in order to determine if distributed switches can be used in the environment.

Consider using distributed switches if possible.

Network Design

Blade Module	VMNIC#	Connection Type
A0	VMNIC0	Service Console
A1	VMNIC1	VMotion
A0	VMNIC2	Fault Tolerance
A1	VMNIC3	VM Production
B0	VMNIC4	Mgmt Spare
B1	VMNIC5	VM Production

------------------------------

In order to accomplish this, here is how we setup it in VMware.

1. We created 2 Distributed Virtual Switches - a production one, and a Service Console One.

2.Under the Service Console switch create a port group for Fault Tolerance, this should be a very limited vlan or for administrative purposes only.

3. Under the Service Console switch create a port group for the Service Console, this should be a very limited vlan or for administrative purposes only.

4.Under the Service Console switch create a port group for Vmotion, this needs to only be on the virtual switch or across any switch ports that Vmotion would be used across. It is something the ESX hosts handle internal to VMware so it doesn't even need to be routable.

5. On the production Switch current port groups as needed for your environment.

Teaming and Failover Setup

1. You need to setup Teaming and Failover to make the uplinks redundant which can be done after you have added the vmnics to the uplinks.

2. Once that is completed edit the settings of each port group. Go to the Teaming and Failover section under policy.

3. Change the links based on the Chart at the top so the uplink that matches the vmnic above is the Active Uplink. Move the rest of the uplink to standby. See the screen shot below for an example.

When all this is done, the ESX host specific configuration, should look like the image below. If your switching configuration is setup correctly then VMware should be now be setup redundant and according to the Health Check best practices.

Vmware ESX Host Drive Space Best Practice

We recently had Vmware technicians come in to analyze our environment we had configured and tell us what needed changed and give us best practices for VMware in our environment. It was through Dell and ultimately there is a Virtualization Healthcheck tool that will go through and tell you the majority of this. That tool runs as a Guest OS host in your VMware enviroment.

This particular post lays out what optimum settings for virtual disk size when setting up and ESX 4.0 host. This is one setting that needs to be done before you go much further since changing it pretty much requires a rebuild of the ESX host.

Here are the settings we ended up using.

• /boot - ext3 1100MB (PreConfigured Space for upgrades)
• - swap 1600+MB (Change for maximum services console swap)
• / - ext3 16384 MB (Change for additional space in root)
• /var - ext3 8192 MB (Create this partition to avoid overfilling root with log files)
• /tmp - ext3 8192 MB (Create this partition to avoid overfilling root with temp files)
• /opt - ext3 8192 MB (Create this partition to avoid overfilling root with VMware HA log files)
• - vmkcore 100MB (Memory Dump for PSOD - Note I didn't have this option)
• Leave all remaining space on the local volume unpartitioned

That is the first major setup best practice that isn't immediately obvious by going through the steps, more to follow including network setup and reasoning since that is one of the toughest ones in our experience.

Reconnecting VMhost after losing connections due to config problems

In trying to figure out our VMware setup, configure the switches and virtual switches many times I ended up losing connectivity to the vmhosts. The majority of the time this was caused by moving the vmnics into and out of the different switches or switch groups.

When ever this occurs the only way I have found to correct it is by going to the host console, for us this is accomplished via the remote console. When using the remote console I have found the following commands to be a great deal of help.

Many of these were taken from various VM sites including here, here or here.

To view the current configuration you can use

esxcfg-vswitch -l

which will create an output like.

For a virtual switch you can use commands like

esxcfg-vswitch -U {$vmnicid#} {$virtualSwitchName} this will remove or unlink the vmnic from the switch.

To link or add a vmnic to a virtual switch use the command

esxcfg-vswitch -L {$vmnicid#} {$virtualSwitchName}

To link or add a vmnic to a Distributed virtual switch or DVP use the command

esxcfg-vswitch -P {$vmnicid#} -V {$dvportId} {$dvswitchName}

To unlink or remove a vmnic from a Distributed virtual switch or DVP use the command

esxcfg-vswitch -Q {$vmnicid#} -V {$dvportId} {$dvswitchName}

Understanding VMware Virtual Nics to Physical Nics on a Dell M1000e

Hi,

After a fair amount of googling and no good answers as to how to the physical to virtual nic configuration between VMware and our Dell M1000e chassis, the blades and the M6220 power connect switches work we made a call to support at Dell and got our questions answered. This post is going to highlight and try and re-explain what Michael Daniels in the alternate OS support explained.

In figure 1 below you can see the view from the dell chassis management. The 3 blades we are working with are the ones I have identified as ESX1, ESX2 and ESX3. Notice they are full height blades this has an impact when it comes to the number of NICS that are available on each blade. Each one of these 3 blades has 1 ethernet mezzanine card installed in the B fabric slot and 1 Fiber card installed in the C fabric slot.

Figure 1.

Moving on to look at the details of the blade in slot 1 you can see I have it selected and currently have the properties of that blade displayed. The important thing to note in this are the MAC addresses and how they relate to the fabrics. This is what you will need if you need to make a one to one correlation of physical blade ports to virtual nics. From this view you can also tell the since it is a full height blade it has ports tied to both slot 1 and slot 9 on the chassis. VMware sees no distinction in these ports, they are simply more virtual nics.

Figure 2

In figure 3 below, which I apologize for being so little, you can the network adapter configuration of what I called ESX1. This view shows the mac addresses that each virtual nic is connected to. These mac addresses will match up with the mac addresses in figure to which is how you can identify what nic goes to what fabric.

Figure 3.

Based on this pictures and comparing MAC addresses we can create the following list of what ports connect to where.

VMware adapter Fabic MAC

vmnic0 A1-slot1 00:21:9B:FE:32:F8

vmnic1 A2-slot1 00:21:9B:FE:32:FA

vmnic2 A1-slot9 00:21:9B:FE:32:FC

vmnic3 A2-slot9 00:21:9B:FE:32:FE

vmnic4 B1-slot1 00:10:18:3B:83:2C

vmnic5 B2-slot1 00:10:18:3B:83:2E

As an additional help in visualizing how things work figure 4 is a screenshot of the Dell Powerconnect M6220. This particular switch is the one in the A1 fabric. From this view you can see the internal and external ports on the switch. Our ESX1 server would have vmnic0 mapped to port g1 and vmnic2 mapped to port g9.

Figure 4

Hopefully this article helped to clear up and answer what virtual nic relates to what physical nic in VMware. Coming soon I will discuss the process we used to configure the ports and vlans in the switches and how based on that to setup a distributed switch in VMware with ESX hosts attached.