Using Native Device Drivers in VMs¶

SmartIO enables passthrough of pooled devices, so they can be used in VMs with near-native performance. Passthrough devices appear as physical devices in guests and can be used with native device drivers without any guest modifications. This section describes how to use Device Lending with virtualization support to pass a pooled device through to a guest running on a cluster node hypervisor.

Warning

Passthrough of pooled devices to VMs is an experimental feature. Please be aware that there might be unresolved bugs and missing features in the support for passthrough of pooled devices.

System Requirements¶

The hypervisor / borrower must be a supported platform.
The hypervisor / borrower must run a supported Linux distribution.
The hypervisor / borrower must have a large enough NTB prefetchable size to map the BARs of the devices to be borrowed.
The lender must have a large enough NTB prefetchable size to map a DMA window large enough to cover the memory of the virtual machine.
Both hypervisor / borrower and lender must have IOMMU / VT-d enabled.
The device must have a driver compatible with virtualization.

Supported kernels¶

SmartIO with virtualization is supported on all hardware platforms where SmartIO is supported, but only on the following Linux kernels.

Linux kernel	Status	Notes
`v6.9` and newer	Unsupported	Will be supported in the future
`v6.8`	Supported	Supported from eXpressWare 5.26
`v5.16` - `v6.7`	Not supported	May be supported upon request
`v4.10` - `v5.15`	Supported	Supported from eXpressWare 5.25
`v4.9` and older	Not supported	May be supported upon request

In addition to the kernel version constraints, your kernel must have the vfio and mdev kernel modules.

Passthrough of pooled devices¶

Hint

Passthrough of pooled devices is a special case of Device Lending. Please refer to Using Native Device Drivers for a comprehensive guide to Device Lending.

When passing through a pooled device to a guest operating system, the SmartIO driver from eXpressWare must run on the hypervisor, and the pooled device must be available for borrowing. Ensure the required kernel modules are running:

$ sudo modprobe vfio
$ sudo modprobe mdev
$ # dis_kosif_vfio.ko must be loaded before the SmartIO module (dis_sio)
$ sudo systemctl stop dis_sio
$ sudo insmod /opt/DIS/lib/modules/$(uname -r)/dis_kosif_vfio.ko
$ sudo systemctl start dis_sio

Enable the device for passthrough with smartio_tool enable-pt. Enabling passthrough for a device does not affect bare-metal borrowing or other cluster nodes.

$ sudo smartio_tool list
48100: VGA compatible controller NVIDIA Corporation Device 28b0 [available]
48101: Audio device NVIDIA Corporation Device 22be [available]
$ sudo smartio_tool enable-pt 48100

We are now ready to start the VM using QEMU. Create a VFIO device in QEMU using the VFIO handle from SmartIO. The pooled device will be borrowed during VM boot.

$ qemu-system-x86_64 \
    -machine q35 \
    ... \
    -device vfio-pci,sysfsdev=/sys/bus/mdev/devices/$(smartio_tool vfio-handle 48100)

If your QEMU distribution supports it, you may also hot-plug the device by following this guide.

Note

Only QEMU’s q35 machine architecture is supported with SmartIO virtualization.

When the VM is terminated or exited, or the device is PCIe hot-unplug, the passthrough device will automatically be put back into the device pool, and can immediately be used by other tenants or bare-metal hosts.