Networking (Standard and Distributed)

This section will cover both standard networking and distributed networking (vCenter server required for distributed networking), first I will start with standard networking.

Standard Networking

The VMkernel can create virtual switches called vSwitches, the virtual machines virtual NIC's are plugged into vSwitches these are then mapped to the physical NIC's on the ESXi server. This means many virtual machines can use the physical NIC's on the ESXI server. The other clever thing VMware can do is that if two virtual machines communicate with each other on the same vSwitch no physical network traffic is generated. The VMkernel moves the data into memory seamlessly from one virtual machine to another without ever involving the physical network interface. Knowing this knowledge you can design your VM's that communicate with each other to use the same vSwitches thus increasing performance.

vSwitches are VLAN aware and can control outbound traffic using a VMware technology called traffic shaping, you can also impose security settings. A vSwitch can contain 0,1 or many physical NIC's assigned to it, a vSwitch without any NIC's attached to it is called a internal vSwitch, as it only allows communication with the host. The internal vSwitch could be used as staging area before moving it into Production, one note to remember is that vSwitches do not communicate with one another.

The only drawback of internal vSwitches is that you cannot carry out vMotion events without first disconnecting users from the virtual machine. Since they are internal to the ESXi server, we cannot guarantee that users would have a continuous connection to the virtual machine during the vMotion event.

A vSwitch with one physical NIC gives you basic connectivity to the outside world. This might be suitable for vMotion or one that does not require fault tolerance (FT), ideally this would be for testing, development environments, etc. A vSwitch with two physical NIC's gives you fault tolerance (FT) and load balancing, this would be idea for vMotion and could be used in Production environments.

You can have up to 20 physical NIC's in an ESXi server of any link speed, there are now 56 ports by default on a vSwitch and this is configurable for up to 4,088 ports.

Port Groups

vSwitches can be divided into smaller units called port groups, there are three types of port groups

You could create one big fat vSwitch and connect all the NIC's to this but you should really separate the network traffic on separate NIC's. Try and name your port groups with a meaningful name so that if support up the service they can see what the port group is used for, idea names would be vlan25, vlan26. An important note to remember is that if you rename a port group virtual machines become orphaned from the switch because the name of the virtual switch to which the virtual machine is attached is held in the virtual machines configuration file (.vmx), so bear that in mind if you rename a port group that many have many virtual machines attached, that's a lot of work to resolve the problem and many even need a script to resolve it. An example of this that I change my iSCSI network port group to iSCSI networks (additional s in network) and as you can see in the image on the left had 6 virtual machines attached, a simple name change now has orphaned all 6 virtual machines, see the image on the right (the images were taken from the summary screen of the ESXi server)

Here is a guide on how to create the various standard vSwitch

Internal standard vSwitch

To create a internal vSwitch follow the notes below, remember to name your vSwitch appropriately

  1. In vSphere Client, select the ESXi host
  2. Select the configuration tab
  3. In the hardware panel, select Networking
  4. Click the Add Network link
  5. Choose Virtual Machine and click next
  6. Make sure no network adapters are selected and then click Next
  7. In the Port Groups Properties dialog box, type a descriptive and unique name as internal0-vmware1
  8. Click Finish

You should end up with something like below

Teamed Standard Switch

To create a teamed vSwitch it is the same as above, apart from selecting two or more physical adapters, you should end up with something like below

Standard vSwitch with VLAN support
ESXi supports two main methods of enabling access to VLAN's
  • External switch tagging (EST) - simply plug in the relevant NIC's to the relevant VLAN's and set the virtual machines IP settings for that network. This will consume a lot of NIC's for each VLAN you need one NIC.
  • Virtual switch tagging (VST) - the network interfaces are plugged into truck ports on the physical switch, truck ports allow many VLAN packets to traverse them, even with just one or two NIC's, ESXi can allow multiple virtual machines to access many VLAN's

To create a VLAN vSwitch follow below

  1. In vSphere Client, select the ESXi host
  2. Select the configuration tab
  3. In the hardware panel, select Networking
  4. Click the Add Network link
  5. Choose Virtual Machine and click next
  6. Select the desired NIC's
  7. put a friendly name and the VLAN ID
  8. click Next and then click finish
  9. to add more VLAN ID's simply select properties
  10. select the add button and repeat the process as above

Hopefully you should have something like below

VMKernel standard vSwitch for vMotion

When you create a vMotion port group you will be asked for the following details

  • IP Address
  • Subnet Mask
  • Default gateway (this is optional)

You can use vMotion across routers using the gateway but this is not a preferred way, normally ESXi servers are attached to the same network to increase performance and other problems.

To create a vMotion vSwitch follow below

  1. In vSphere Client, select the ESXi host
  2. Select the configuration tab
  3. In the hardware panel, select Networking
  4. Click the Add Network link
  5. Choose VMKernel and click next
  6. select the desired NIC's
  7. In the port groups dialog box type a friendly name, in my case I type "vMotion"
  8. The select the "use this port group for vMotion"
  9. Set the IP address and subnet mask and gateway if required

Hopefully you should get something like below, you may get a license warning come up, it still will create the port group.

Configuration and Problems of vSwitches and Port Groups

To check to see if a NIC has a problem connecting to the network, go to the Networking screen and check if there is a red-cross against the NIC, see the below image, use standard networking fault finding to correct the problem.

There are a number of configuration settings on vSwitches and Port Groups

Increase the number of ports on a vSwitch

You can now change the number of ports up to 4088 with ESXi 4, if you have a large number of VM's you may have to increase this value

To increase the number of ports on a vSwitch

  1. In vSphere Client, select the ESXi host
  2. Select the configuration tab
  3. In the hardware panel, select Networking
  4. Click the properties of the vSwitch that you wish to change
  5. make sure the vSwitch option is highlighted
  6. select the edit button
  7. then using the down arrow, select number ports that you desire
  8. click OK to implement


Setting speed and duplex on physical NIC's

You can change the speed and duplex of any on the physical NIC's in your ESXi server, generally they will auto-negotiate

  1. In vSphere Client, select the ESXi host
  2. Select the configuration tab
  3. In the hardware panel, select Networking
  4. Click the properties of the vSwitch that you wish to change
  5. select the "Network Adapters" tab at the top
  6. select the physical NIC and click the edit button
  7. choose the desired speed and duplex
  8. click OK to implement


Setting vSwitch and Port Group policies

I am going to cover these briefly, you have three additional areas that can be changed

  • promiscuous mode - allows a NIC to collect all network packets including ones that are not intended for it
  • MAC address change - allow for the MAC address to change, this may be a requirement for things like Microsoft cluster and loading balancing
  • Forged Transmits - allows a VM to send traffic under a MAC address that is different from that of the VM, Microsoft Network Loading balancing may break if you reject this

Most systems the default option will be fine

Traffic Shaping

Traffic shaping is ESXi method of controlling outbound traffic generated by VM's you can control the bandwidth, traffic shaping is not dynamic once set those values are set in stone. Not sure why you would use this feature as i like to give my servers as much bandwidth as possible but you can make use of it.

NIC Teaming

NIC teaming (bonding in the Linux world) is a fault tolerance, load balancing feature, you can configure the load balancing, what to detect to cause a failover, failback when the problem has been solved and select particular NIC's to standby or active NIC's

This is pretty much the same as all NIC teaming in other O/S

Service Console Network

ESXi 4 the networking architecture treats the Service Console as if it were just another VM connected to a vSwitch, this default vSwitch is called vSwitch0, the service console ports have a special name vswif which stands for virtual switch interface. You should try to have a backup service console port or try to protect the existing by making use it is fault tolerant to network failures. If you were to lose the service console to have to go to the command-line to either re-install it of repair it. Here are some commands that will help in recovering a failed service console

Service Console troubleshooting
Correcting your NIC selection

# Displaying the vSwitch configurations

  esxcfg-vswitch -l

# Display all the NIC's

  esxcfg-nics -l

# Link another NIC to the switch

  esxcfg-vswitch -L vmnic1 vSwitch0

# Unlink a NIC from a Switch

  esxcfg-vswitch -U vmnic0 vSwitch0

Correcting IP settings

# Display current IP address settings

  esxcfg-vmknic -l

# Change the Service console network

  esxcfg-vmknic "Management Network" -i -n

# Restart the network services

  service network restart

Configuring the Cisco Discovery Protocol

# Display current CDP settings, it should return "listen"

  esxcfg-vswitch -b vSwitch0

# Make the vSwitch bidirectional

  esxcfg-vswitch -B both vSwitch0

# Display CDP data

  esxcfg-info | more +/CDP\ Summary

Change the MTU value

# Change the MTU value

  esxcfg-vswitch -m 1500 vSwitch0

Distributed Switches

Distributed virtual networking (DVN) is completely new to version 4, it enhances the network layer of the VMkernel, in uses features such as below, however you must be using vCenter in order to create DvSwitches

At a basic level a DvSwitch is a global switch, rather than configure vSwitches for each ESXi server, you create a DvSwitch that can be applied to every ESXi server, they also implement private VLAN's (PVLAN) within VMware. DvSwitches can do all the things a normal vSwitch can do, they also offer more ports than a standard vSwitch.

DvSwitches use distributed virtual uplink (DvUplink) ports, it is merely a container for holding a reference to a physical NIC. When you create a DvSwitch the system attempts to set the correct number of DvUplink ports for you, it does this by looking at all the ESXi servers and searching for the ESXI server with the most uplinks. DvUplinks can be renamed this allows you given more meaningful names.

I am going to combine a number of items to explain how to create and configure a DvSwitch, I am going to create a DvSwitch, add a port group and then add another port group that uses vMotion. This should give you a fairly good understanding of DvSwitches, I will finish off looking at the advanced features (actually they are pretty much the same as a standard vSwitch).

First let me explain what I had already configure, I had a vSwitch configured with two port groups one for the private LAN and one port group for vMotion (don't worry too much about vMotion for the moment I will be covering this in greater detail in a later section).

Current setup and freeing up some NIC's

This what I had setup before, a vSwitch with two port groups. Each ESXi server I have (vmware1 and vmware2) has this vSwitch setup.

The first thing I do is free up some NIC's by removing the Physical adapters from both ESXi servers, you should end up with something like below, you can keep the groups in case you want to restore back to what you had.

Creating a DvSwitch

To create a DvSwitch first go to the home page in the vCenter and select networking. You can either select the "New vNetwork Distributed switch" icon or select the "Add a vNetwork Distributed Switch" in the main screen, you then get the the below screen

Select vNetwork Distributed Switch Version 4.1.0 (unless you need version 4.0) and click Next

I changed the name to "Private DvSwitch" but you can call it whatever you want, because I have only one NIC to offer from each of my ESXi servers (vmware1 and vmware2), I change the "Number of dvUplink ports" to one. In the real world you probably will have many NIC's to add so this can be increased, click next when finished.

Expand the ESXi server which then should display the available NIC's (if you remember these are the ones we free 'ed up earlier). Select all NICs and then click next to continue

You now get a summary screen, notice the "automatically create a default port group" is ticked, you can leave this as we will make use of this group. Click next to create the DvSwitch. After renaming the dvSwitch to "dvswitch-NIC2" and renaming the port group to "Private" (you can do this by right-clicking each item and selecting edit settings, then change the name at the top", you should have something like below

If you select the dvSwitch-NIC2 icon you get a number of tabs appear, to view the ports select ports, remember we selected one dvUplink port for each ESXi server which gives us two in total, you can also see that "Connectee" comes from each ESXi server

You can confirm the ESXi servers by then select the "Hosts" tab

So what you have now is a DvSwitch with two NIC's attached (one from vmware1 and one from vmware2), you can makes changes to the NIC's and it will be implemented on both ESXi servers, this can be very handy if you have lots of ESXi servers to manage as most configurations only need slight tuning. The private port group will be empty as we have no VM's using it, I will be discussing how a VM can make use of this in my virtual machine section.

Next we will create a vMotion port group within our DvSwitch, follow below

Create a vMotion port group

Right click on the DvSwitch icon and select "New port group"

I have change the name to "vmotion", but again you can choose whatever you want

Once the port group is created you need to add the vMotion information for each ESXi server, you will need to provide an IP address and the subnet mask for each ESXi server using the vMotion.


So from the Home page select "Hosts and Clusters", from there select the first ESXi server (in my case it is vmware1) and select the configuration tab. Select Networking from the hardware then select the "vNetwork Distributed Switch and you then should have a screen similar to below. Select the "Manage Virtual Adapters" ,then select add at the top

You should then have the screen below, select "New Virtual adapter"

then select "VMKernel" (this was the only choice for me)

Now select the vmotion port group from the first drop down list and also tick the "Use this virtual adapter for vMotion"

Now enter the IP address and subnet mask, when you repeat this step for the other ESXi server remember you use a different IP address but keep it on the same subnet

Next comes the summary screen, as you can see the vMotion port group now has a new VMKernel port with the IP address that you choose


Once you have configured all the ESXi servers, if you select from the "home page" -> networking, then select the DvSwitch group icon, then select the configuration tab you will see the screen below. I have two physical NIC's from the ESXi servers and two IP address for the vMotion (one each for each ESXi server). If you click on one of the IP addresses you can see what NIC it goes to (follow the orange line), as you can see IP address goes to vmnic2 vmware1, in this case. Also from this screen if you look in the top right hand corner you can manage the DvSwitch by removing it, adding hosts, managing hosts (use this option to add additional NIC's), creating a new port group and evening editing the existing DvSwitch.

You may have noticed that I have a VM using the private port group, I will cover this in greater detail in my virtual machine section.

Create a port group for the Service console

This pretty much the same when you get to the "add Virtual Adapter" screen, I choose the service_console port group (this was a new port group that I added) and then ticked the last option "Use this virtual adapter for management traffic, you will then be ask to enter an IP address and subnet for the service console

Once you have finished the screen should look like below see my new service console port group.

As you see a DvSwitch is very similar to a normal standard vSwitch, have a look at each of them side by side, NIC's are represented on the right-hand side and the port groups are on the left-hand side, there is very little difference in them.

Standard vSwitch

I am not going to discuss how to remove a DvSwitch as it is in reserve order as creating one, first remove from the VMKernel ports from each ESXi server first (otherwise when you try to remove the DvSwitch it will say it's in use), then remove each port group and lastly remove the DvSwitch. Most of the time you remove the item by selecting it's icon and right-clicking, then selecting remove.....

The last topic to cover is the advanced sections of a DvSwitch, again there is very little difference than a normal vSwitch

Add/Remove NIC

You can add or remove additional NIC's form the DvSwitch view. From the home page select networking, then select the DvSwitch icon then select configuration from the tabs, you should see a screen like below

Select manage hosts from the top right-hand corner, then select the hosts from which you wish to add NIC's to then select next, you should see the hosts and available NIC's

Select the additional NIC's and select next, if the NIC's already belong to an existing port group it will ask you if you wish to migrate them, as you can see the one i select already belong to an existing group hence the warning message. Continue on and the NIC will be added to your DvSwitch.

To remove a NIC just deselect at the "Select Physical Adapters page" and continue through the rest of the screens.


Advanced settings for a DvSwitch

You can edit the setting of a DvSwitch by right-clicking it's icon and then select "edit settings", the first tab called properties you can change the name, increase/decrease the number of dvUplinks and even put notes for anyone.

The next tab "Network Adapters" just displays the available ESXi servers and the available dvUplinks

The last tab "private VLAN" you can add primary or secondary VLAN ID's

Advanced settings for the DvSwitch adapters

I am not going into much detail on this as it is pretty much the same as the standard vSwitch

  • Security - covers the promiscuous mode, MAC address change and the forged transits
  • Traffic shaping - covers Ingress traffic (traffic that comes from outside a ESXi server - inbound traffic management) and Egress activity that comes into the ESXi server from either virtual machines or the ESXI server itself
  • VLAN - select the VLAN type and VLAN ID's
  • Teaming and Failover - the same as a standard vSwitch but the NIC references are related to dvUplinks port values
  • Miscellaneous - allows you to enable port blocking
  • Advanced - allow override port polices (override the settings on the dvUplink group) and Configure reset at disconnect

Final Comments

DvSwitches are idea for large corporations, it can be a headache to manage lots of standard vSwitches, but bear in mind that you do require vCenter to use DvSwitches which comes with a price and can be off putting to smaller corporations.