How To VMware VMDirectPath (PCI Passthrough)

How To setup VMware ESXi VMDirectPath (aka PCI Passthrough)

VMware's PCI Passthrough solution is by far the best I have used.  The other virtualization platforms (eg. Microsoft Hyper-V, Xen, Citrix XenServer, Oracle VM, KVM, etc) provide little, or no, PCI Passthrough support.  I have found that ESXi 5.0 had the best PCI Passthrough support, so stick with that version.  (ESXi 5.1 was not quite as stable in this area)

VMDirectPath I/O Summary

VMDirectPath I/O, aka PCI Passthrough, is a method of giving a VM direct access to a PCI device.
This VMware KB article describes configuring a device for PCI Passthrough:

• Configuring VMDirectPath I/O pass-through devices on an ESX host - http://kb.vmware.com/kb/1010789

As there is direct access to the PCI device, the only setup required is to create a VM and then assign the desired PCI device to the VM, using the direction in the KB article. No driver need be installed on the ESX host, only within the VM's guest OS.  To configure the VM follow the standard procedures for the target operating system, as though it were a physical system.

Requirements

The server's CPU must support and and BIOS must have enabled: "Intel® Virtualization Technology for Directed I/O (VT-d)" or "AMD I/O Virtualization Technology (IOMMU)"

Limitations

• Not all ESX servers have full support for VMDirectPath, and on some servers not all PCIe slots do either. (especially the desktop class systems)
• The server CPU/BIOS must support Intel VT-d or AMD IOMMU.
• Limited to a maximum of 8 pass through devices powered on simultaneously, per host
• Limited to passing through a maximum of 6 devices per VM

WARNING: When updating firmware through VMDirectPath, when fio-update-iodrive requests a reboot, this means a reboot of the physical hardware, not just the VM.

VMDirectPath Procedure

Note: These same procedures work for ESX/ESXi 4.x and ESXi 5.x.

1. Export out PCI devices through "Advanced Configuration".  The names may not be what you would recognize, so it is best to match on "Device ID" and "Vendor ID".

2. Reboot ESX/ESXi server.

3. Add PCI Device to VM's configuration.

4. Set Memory Reservation.  PCI Passthrough requires all allocated memory to be reserved for the VM.  Set the VM's Memory Reservation to match the VM's Memory Allocation.  Simply drag the Memory Reservation slider all the way to the right to the orange triangle.  This will also need to be done anytime the memory, for the VM, is adjusted in the future.

5. Treat device, in VM, as you would a physical server (install the OS specific driver within the target VM).