Sunday 28 October 2012

Single Root I/O Virtualization (SR-IOV) - Configuration


Here I am detailing the configuration steps and the caveats faced during implementation. 

1. Upgrade your Server BIOS and NIC Firmware to a version supporting SR-IOV. For Dell PowerEdge R710, minimum BIOS is 6.2.3 and 82576 Intel NIC Firmware is 13.5. Refer to my previous blog "Upgrade Dell PowerEdge Components" to understand how to upgrade server components. 
2. From the BIOS, enable Intel VT-d and SR-IOV Global.
 
3. Identify the driver for the NIC that will be used for SR-IOV (in VMware terms its called kernel module). This can be identified from vSphere Client or using CLI (I prefer this method since it provides more details).

From vSphere Client,
a. Navigate to Home > Inventory > Hosts and Clusters > #Select the host# > Configuration Tab > Networking.
b. In case of standard switch, select Properties > Network Adapters Tab. In case of distributed switch, select Manage Physical Adapters.
c. Select the adapter to be used for SR-IOV. On the right pane you will get all the details for the NIC.
From CLI,
a. SSH to the server
b. List all the NICs available on the server
c. Identify the NIC of interest and get more details about it. Remember this NIC should be one of the listed models which support SR-IOV. Refer to the previous blog to see the supported NICs.

~ # ethtool -i vmnic5
driver: igb
version: 2.1.11.1
firmware-version: 1.5-1
bus-info: 0000:06:00.1

This NIC supports SR-IOV as listed in Intel and VMware (Intel Corporation 82576 Gigabit Network Connection).

4. The driver version for vmnic5 is 2.1.11.1. This version doesn't support SR-IOV which is listed in VMware support list as well as Intel support list. You need to upgrade the driver of your NIC. For Intel 82576, I have upgrade to version 3.4.7.3. I will brief how to upgrade the driver.

a. Download the driver from VMware download center based on the version of ESXi (in my case it was VMware ESXi 5.0 Driver for Intel 82580 and I350 Gigabit Ethernet Controllers)
b. Unzip the driver file and identify a file named as #driver_name#-#version#-offline_bundle-#build_number# (in my case it was igb-3.4.7.3-offline_bundle-804663).
c. From vSphere Client, navigate to Home > Solutions and Applications > Update Manager > Patch Repository.
d. Select Import Patches hyperlink and select the file from your machine (igb-3.4.7.3-offline_bundle-804663).
e. Create a new Baseline with type as Extension and include your driver in this baseline.
f. Navigate to Home > Inventory > Hosts and Clusters > #Select the desired Host# > Update Manager Tab.
g. Attach the baseline to your host and Stage/Remediate.

You can verify the update by logging into CLI.

~ # ethtool -i vmnic5
driver: igb
version: 3.4.7.3
firmware-version: 1.5-1
bus-info: 0000:06:00.1

Also, you can update the driver using CLI instead of VUM.

5. Create a Host Profile for the host which has the NIC to be used for SR-IOV. To create the profile Right-Click the Host > Host Profile > Manage Profile. 
6. Navigate to Home > Management > Host Profiles. Select the interested profile and Edit Settings. 
7. From the Settings Navigate to Kernel Module Configuration > Kernel Module > #Driver_Name# > Kernel Module Parameters > max_vfs > Module Parameter Settings.
Note: In case your driver doesn't support SR-IOV, max_vfs kernel parameter won't be listed under the driver. This is another method to check your driver support to SR-IOV.

Enter the max number of Virtual Functions (VFs) per Physical Function (PF). In case of dual-port 82576, you can enter the value as max_vfs=x,x. The max number of VFs per PF depends on the NIC. For Intel 82576 it is 16.

Note: A value of '0' means that SR-IOV is disable for the specified PF.

8. Apply the profile to the host after the modifications and reboot the host. 
9. Once the host is up, Select the host and navigate to Configuration > Network Adapters. The NICs enabled for SR-IOV won't be listed. Navigate to Advanced Settings under Hardware section, you will see that VFs are listed as Passthrough devices. At this stage, you have completed SR-IOV configuration.  
10. Shutdown you VMs > Edit Settings > Add PCI Devices. You VM Guest OS should be supporting SR-IOV and having VFs drivers.

4 comments:

  1. The max_vfs=x,x parameter will enable x number of vfs per port that the ixgbe or igb driver owns. So if there are four 1Gb ports and if you want 7 vfs on the second port in the PCIe list then use 0,7,0,0. Note: that even though the controller support 8vfs on 1Gb and 64vfs on 10Gb from Intel only N-1 vfs can be enabled per port.
    The maximum VFs per port for Intel Ethernet Controllers that support SR-IOV:

    Intel® Ethernet Controller X540 Family (63 per port)
    Intel® Ethernet Controller X540-AT2
    Intel® Ethernet Converged Network Adapter X540 Family (63 per port)
    Intel® Ethernet Converged Network Adapter X540-T1
    Intel® Ethernet Converged Network Adapter X540-T2
    Intel® 82599 10 Gigabit Ethernet Controller Family (63 per port)
    Intel® Ethernet 82599EB 10 Gigabit Ethernet Controller
    Intel® Ethernet 82599ES 10 Gigabit Ethernet Controller
    Intel® Ethernet 82599EN 10 Gigabit Ethernet Controller
    Intel® Ethernet Converged Network Adapter X520 Family (63 per port)
    Intel® Ethernet Server Adapter X520-DA2
    Intel® Ethernet Server Adapter X520-SR1
    Intel® Ethernet Server Adapter X520-SR2
    Intel® Ethernet Server Adapter X520-LR1
    Intel® Ethernet Server Adapter X520-T2
    Intel® Ethernet Controller I350 Family (7 per port)
    Intel® Ethernet Controller I350-AM4
    Intel® Ethernet Controller I350-AM2
    Intel® Ethernet Controller I350-BT2
    Intel® Ethernet Server Adapter I350 Family (7 per port)
    Intel® Ethernet Server Adapter I350-T2
    Intel® Ethernet Server Adapter I350-T4
    Intel® Ethernet Server Adapter I350-F2
    Intel® Ethernet Server Adapter I350-F4
    Intel® 82576 Gigabit Ethernet Controller Family (7 per port)
    Intel® 82576EB Gigabit Ethernet Controller
    Intel® 82576NS Gigabit Ethernet Controller
    Intel® Gigabit ET/EF Server Adapter Family (7 per port)
    Intel® Gigabit ET Dual Port Server Adapter
    Intel® Gigabit EF Dual Port Server Adapter
    Intel® Gigabit ET2 Quad Port Server Adapter

    ReplyDelete
    Replies
    1. Thx mate for your comment. Yes you are right. For Intel® 82576 Gigabit Ethernet Controller Family (7 per port).

      Its a typo mistake since the max number of VFs for Emulex is 16 per port.

      Delete
  2. Excellent work and superb collection of facts. People may think about Search Engine Optimization too.
    Server hosting 1gbit

    ReplyDelete
  3. Baqari, where did you get firmware update for 82576 Intel NIC Firmware?

    I have the same server Dell PowerEdge R710 just as you have. I already searched for firmware update on dell support site with no success...

    ReplyDelete