Passthrough of non-SR-IOV pooled devices to unmodified guests¶
Hint
Passthrough support is scheduled for the upcoming 5.25 release. Please contact support for early access.
SmartIO enables passthrough of all pooled devices running as paravirtualized devices with near-native performance. Passed-through devices appear as physical devices in guests and can be used with native device drivers without any guest modifications. This section describes how to use Device Lending with virtualization support to pass a pooled device through to a guest running on a cluster node hypervisor.
System Requirements¶
Warning
Passthrough of non-SR-IOV pooled devices is currently only supported for linux
kernel v4.10 - v5.1. This will be expanded in future eXpressWare releases.
See Supported kernels platforms for details.
The hypervisor / borrower must be a supported platform.
The hypervisor / borrower must run a supported linux distribution.
The hypervisor / borrower must have a large enough NTB prefetchable size to map the all BARs of the devices to be borrowed.
The lender must have a large enough NTB prefetchable size to map a DMA window large enough to cover the memory of the Virtual Machine / guest.
Both hypervisor / borrower and lender must have IOMMU / VT-d enabled.
The device must have a driver compatible with virtualization.
Supported kernels platforms¶
SmartIO with virtualization is supported on all hardware platforms where SmartIO is supported, but is only supported on the following Linux kernels.
Linux kernel |
Status |
Notes |
|---|---|---|
|
Not supported |
Will be supported in the future |
|
Not supported |
Will be supported in the future |
|
Supported |
Supported from eXpressWare 5.25 |
|
Not supported |
Passthrough of pooled devices¶
Hint
Passthrough of non-SR-IOV pooled devices is a special case of Device Lending. Please refer to Using Native Device Drivers for a comprehensive guide to Device Lending.
When passing through a pooled device to a guest operating system, the SmartIO driver from eXpressWare must run on the hypervisor, and the pooled device must be available for borrowing. To prepare a device for passthrough, we must register it as a mediated device. From the borrower, grab the fabric device id of the pooled device:
$ smartio_tool list
48100: VGA compatible controller NVIDIA Corporation Device 28b0 [available]
48101: Audio device NVIDIA Corporation Device 22be [available]
Then, inspect the pooled device with smartio_tool show:
$ smartio_tool show 48100
Name: VGA compatible controller NVIDIA Corporation Device 28b0
Available: available
Location: remote
Adapter: 0
NodeId: 4
Remote BDF: 0000:81:00.0
Physical slot: N/A
Serial Number: 00-00-00-00-00-00-00-00
UUID: 756e81e3-f403-42a3-b848-862013cca6c6
Vendor ID: 10de
Device ID: 28b0
Subsystem Vendor ID: 10de
Subsystem Device ID: 1870
Local users: 0
The UUID of the device must be passed to the Linux mediated device driver (mdev). You can find the UUID using the above command, or through sysfs under proc/smartio/remote/devices/<fdid>/uuid. Next, we pass the UUID to the SmartIO mediated device bus. Look under /sys/class/mdev_bus/<domain>:<bus number>/. In our case, the domain and bus number is 0000:0b.
$ SMARTIO_UUID=756e81e3-f403-42a3-b848-862013cca6c6
$ echo $SMARTIO_UUID | sudo tee /sys/class/mdev_bus/0000:0b/mdev_supported_types/pci_bus-smartio/create
We see from the kernel log buffer that SmartIO has created a mediated child device from the device UUID we passed:
$ sudo dmesg
...
dis_smartio: Created mediated child device from 756e81e3-f403-42a3-b848-862013cca6c6
...
We are now ready to start the unmodified guest using QEMU. Pass the UUID to QEMU with -device vfio-pci,sysfsdev=<uuid>.
$ qemu-system-x86_64 \
... \
-device vfio-pci,sysfsdev=/sys/bus/mdev/devices/$SMARTIO_UUID
You can pass multiple pooled devices by simply adding more arguments:
$ qemu-system-x86_64 \
... \
-device vfio-pci,sysfsdev=/sys/bus/mdev/devices/$GPU1_UUID \
-device vfio-pci,sysfsdev=/sys/bus/mdev/devices/$GPU2_UUID
Note
We recommend running your virtualizer with KVM enabled for best performance.
Returning pooled devices¶
When the guest operating system is exited, the passed-through device will automatically be put back into the device pool, and can immediately be used by other tenants or bare-metal hosts.