Lending out devices to other hosts

Establishing connection to other nodes

To allow other nodes to see and borrow devices from a node, a connection must first be established. The connection is one-way, from the lending side to the borrowing side:

$ smartio_tool connect 8

A connection can be established to multiple nodes, and in both directions if desired. For instance, if node 4 and node 8 want to swap devices, smartio_tool connect must be run on both nodes. A node may also connect to itself. This allows the SmartIO SISCI API to be for local devices as well.

If the connection is interrupted, by for instance a link failure, the connection is re-established when possible. If the borrow-side reboots, the lending side will reconnect when possible, but if the borrow side reboots, the connection must be manually re-established using smartio_tool connect.

Finding and adding devices

Devices that are to be borrowed by other nodes must first be added to the list of lendable devices. This is done using the command smartio_tool add <BDF>. The BDF argument is supplied by the user to specify what device is to be added. The BDF of the device to be added can be found using lspci:

$ lspci
05:00.0 VGA compatible controller: NVIDIA Corporation GK208 [GeForce GT 710B] (rev a1)
05:00.1 Audio device: NVIDIA Corporation GK208 HDMI/DP Audio Controller (rev a1)
06:00.0 Non-Volatile memory controller: Intel Corporation Device f1a5 (rev 03)
07:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03)

In the example above, the BDF of the NVMe drive is 06:00.0:

$ smartio_tool add 06:00.0

After adding a device, the device will be visible to other connected nodes. Added remote devices can be found by running the list command:

$ smartio_tool list
ID 80002: Non-Volatile memory controller: Intel Corporation Device [unavailable]

Added devices are initially unavailable and are shown as such on remote nodes. To make the device available, it must first be set as available on the lending side using smartio_tool available BDF:

$ smartio_tool available 06:00.0

Warning

Making critial devices “available” may crash the system. For example making the system boot drive available will not go well. Consider if the device is currently in use before proceding.

This will unbind the device’s local device driver and bind it to SmartIO’s device driver. This will make the device unusable locally and associated resources will no longer be available. Since the device in this example is a disk drive, we should unmount any mounted partitions before setting it as available. Remote nodes that run smartio_tool list at this point will see that the device is available:

$ smartio_tool list
80002: Non-Volatile memory controller: Intel Corporation Device [available]

To reclaim a device for local use and make it unavailable for remote nodes, the smartio_tool unavailable command is used:

$ smartio_tool unavailable 06:00.0

If any remote node is currently borrowing the device when the device is set as unavailable, it will be taken back. Depending on the type of device, it is recommended that the other side releases it’s use of the device before it’s set as unavailable. This is explained in the section Borrowing and returning remote devices.

Adding devices using filters

Filters can be used to add devices based on various properties. The filters have the same syntax as lspci. This means that the filter can be tested and confirmed using lspci before being passed to smartio_tool. To add all NVMe drives we can use a filter for the class 0108:

$ lspci -s ::0108
06:00.0 Non-Volatile memory controller: Intel Corporation Device f1a5 (rev 03)
$ smartio_tool add-filter -d ::0108
$ smartio_tool available-filter -d ::0108

Devices can also be added by vendor id, device id or a partial BDF. Any combination can also be used:

#Add all devices with vendorid NVIDIA
$ smartio_tool add-filter -d 10de::

#Add all devices on bus 1
$ smartio_tool add-filter -s 1: