Lending Local Devices¶

The CPU and memory of the lender will only be used for initialization and will not be active/used when the borrowing system is using the PCIe devices. All PCIe transactions and system interrupts will be forwarded to the borrowing side by the PCIe hardware.

System Requirements¶

The lender must be a supported platform that supports peer-to-peer.
The lender must run a supported operating system.

Additional System Considerations¶

Check if the device to be lent has any known limitations or quirks.
Consider setting a large NTB prefetchable size. The size required depends on how the device is borrowed.
On NUMA systems, consider placement of device and NTB host adapter. See NUMA and Multi-Socket Systems.
Consider the tradeoffs for isolation vs performance for IOMMU / VT-d.

Establishing Connection¶

To allow other nodes to see and borrow devices from a node, a connection must first be established. The connection is one-way, from the lending side to the borrowing side:

$ smartio_tool connect 8

A connection can be established to multiple nodes, and in both directions if desired. For instance, if node 4 and node 8 want to “swap” devices, smartio_tool connect must be run on both nodes. A node will automatically be connected to itself.

If the connection is interrupted, by for instance a link failure, the connection is re-established when possible. If the borrow-side reboots, the lending side will reconnect when possible, but if the lender reboots, the connection must be manually re-established using smartio_tool connect.

Adding Devices¶

Devices that are to be borrowed by other nodes must first be added to the list of lendable devices. This is done using the command smartio_tool add. The BDF argument is supplied by the user to specify what device is to be added. The BDF of the device to be added can be found using lspci:

$ lspci
00.0 VGA compatible controller: NVIDIA Corporation GK208 [GeForce GT 710B] (rev a1)
00.1 Audio device: NVIDIA Corporation GK208 HDMI/DP Audio Controller (rev a1)
00.0 Non-Volatile memory controller: Intel Corporation Device f1a5 (rev 03)
00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03)

In the example above, the BDF of the NVMe drive is 06:00.0:

$ smartio_tool add 06:00.0

After adding a device, the device will be visible to other connected nodes. Added remote devices can be found by running the list command:

$ smartio_tool list
ID 80002: Non-Volatile memory controller: Intel Corporation Device [unavailable]

Added devices are initially unavailable and are shown as such on remote nodes. To make the device available, it must first be set as available on the lending side using smartio_tool available BDF:

$ smartio_tool available 06:00.0

This will unbind the device’s local device driver and bind it to SmartIO’s device driver. This will make the device unusable locally and associated resources will no longer be available. Since the device in this example is a disk drive, we should unmount any mounted partitions before setting it as available. Remote nodes that run smartio_tool list at this point will see that the device is available:

$ smartio_tool list
80002: Non-Volatile memory controller: Intel Corporation Device [available]

To reclaim a device for local use and make it unavailable for remote nodes, the smartio_tool unavailable command is used:

$ smartio_tool unavailable 06:00.0

If any remote node is currently borrowing the device when the device is set as unavailable, it will be taken back. Depending on the type of device, it is recommended that the other side releases it’s use of the device before it’s set as unavailable.

Filters¶

Filters can be used to add devices based on various properties. The filters have the same syntax as lspci. This means that the filter can be tested and confirmed using lspci before being passed to smartio_tool. To add all NVMe drives we can use a filter for the class 0108:

$ lspci -s ::0108
06:00.0 Non-Volatile memory controller: Intel Corporation Device f1a5 (rev 03)
$ smartio_tool add-filter -d ::0108
$ smartio_tool available-filter -d ::0108

Devices can also be added by vendor id, device id or a partial BDF. Any combination can also be used:

# Add all devices with vendor id NVIDIA
$ smartio_tool add-filter -d 10de::

# Add all devices on bus 1
$ smartio_tool add-filter -s 1:

Lending Virtual Function of an SR-IOV device¶

With a device that supports SR-IOV you can create one or more virtual functions. The exact functionality provided by the virtual functions depends on the device, but typically the resources provided by the physical function will be split into the configured number of virtual functions. For example, an SR-IOV GPU can provide multiple virtual GPUs with a partition of the physical GPU’s resources. NVMe drives with SR-IOV on the other hand will provide access to the same underlying storage, but this can be further managed using NVMe namespaces. The specifics of how to enable SR-IOV on your device is beyond the scope of this manual. We recommend that you follow the manufactures guide or community guide for creating the virtual functions. Once the virtual functions have been created and are visible in lspci they can be added and made available like any other PCIe device. Note that after virtual functions are enabled, the physical function cannot be made available to other nodes, but must be managed by the lender.