smartio_tool¶

smartio_tool - Low level tool for interacting with SmartIO

Synopsis¶

smartio_tool <command> [options...]

Description¶

smartio_tool interacts with both the lending side and borrow side of the SmartIO module. See below sections for a list of commands for each side.

Lending local devices¶

Commands accepting a BDF expects the same format as accepted by lspci -s. Example 01:00.0. It also accepts the same short-hand notation as lspci when this uniquely identifies a single device, for example: 1:

connect <NODEID> [ADAPTER]¶: Connect to a remote node, allowing it to see and borrow this node’s devices. A connection is also required between two lenders if devices in them intend to perform peer-to-peer transfers. Each lender will have a loopback connection to its own node’s borrow-side.

add <BDF>¶: Add the given device to the device pool. The device is not made available and this has no effect on the device.

remove <BDF>¶: Remove the given local device from the device pool.

available [--force] <BDF>¶: Make the given local device available for borrowing. If the device is currently bound to a device driver, the user will be prompted for confirmation before the device is made available. The --force flag can be used to skip the confirmation. Note that unbinding the local device driver will make the device unavailable for local use. The user must ensure that the device is not in use before making it available. (For example, unmount an NVMe drive or stop applications using a GPU).

unavailable <BDF>¶: Make the given local device unavailable for borrowing. The device must currently be available. This will rebind any local device driver.

add-filter <filter>¶: Add multiple devices using a filter. See Device filters.

available-filter <filter>¶: Mark multiple devices as available. See Device filters.

unavailable-filter <filter>¶: Mark multiple devices as available. See Device filters.

Pool commands¶

list¶: Lists all devices in the pool.

lspci <fdid> [lspci args]¶: Runs lspci command on a fabric device. See manpage for lspci(8).

show <fdid>¶

Show extended information about a fabric device:

$ smartio_tool show 48100
Name: VGA compatible controller NVIDIA Corporation Device 28b0
Available: available
Location: remote
Adapter: 0
NodeId: 4
Remote BDF: 0000:81:00.0
Physical slot: N/A
Serial Number: 00-00-00-00-00-00-00-00
UUID: 756e81e3-f403-42a3-b848-862013cca6c6
Vendor ID: 10de
Device ID: 28b0
Subsystem Vendor ID: 10de
Subsystem Device ID: 1870
Local users: 0

Adding fabric attached devices to the pool¶

scan <node id> [adapter]¶: Scan for and initialize transparent devices not connected to a host. The given nodeid is assigned to the transparent tree, and thus must be unique in the cluster. If the given nodeid is already initialized the tree will be scanned for changes. The adapter parameter specifies the local adapter number to be used. If not given, adapter 0 is used.

Borrowing and using devices¶

borrow <fdid> [dma window size]¶: Borrow a fabric device and inject as a local device. See section :ref:borrow-device.

return <fdid>¶: Undo borrow command

enable-p2p <source-device-fdid> <destination-device-fdid> [BAR]¶

disable-p2p <source-device-fdid> <destination-device-fdid> [BAR]¶

get-vdev <fdid>¶: Print the (virtual) local BDF of the borrowed device.

Advanced commands¶

config <CONFIG> <VALUE>¶: Sets SmartIO kernel module parameter CONFIG to VALUE.

Device filters¶

Some smartio_tool commands operate on a lspci-based filter. These commands will operate on all devices that lspci would list given the same filter. Such filters can be tested by running lspci <filter>. Example:

# lspci -d 10de:
09:00.0 VGA compatible controller: NVIDIA Corporation GK107GL [Quadro K420] (rev a1)
# smartio_tool add-filter -d 10de:
Adding 09:00.0 VGA compatible controller: NVIDIA Corporation GK107GL [Quadro K420] (rev a1)

See manpage for lspci(8) for details.

Examples¶

Adding, setting as available and borrowing a device:

Lending side:

$ smartio_tool connect 4
$ smartio_tool connect 8
$ smartio_tool add 06:00.0
$ smartio_tool list
80002: Non-Volatile memory controller: Intel Corporation Device [unavailable]
$ smartio_tool available 06:00.0
$ smartio_tool list
80002: Non-Volatile memory controller: Intel Corporation Device [available]

Borrow side:

$ smartio_tool borrow 80002 512
Name: Non-Volatile memory controller Intel Corporation Device f1a5
Local users: 1
Local virtual device: 0000:04:05.0
Bound to driver: nvme
NVMe namespace: nvme0
$ ls /dev/nvme0*
/dev/nvme0  /dev/nvme0n1  /dev/nvme0n1p1
$ mount /dev/nvme0n1p1 /mnt
$ umount /mnt
$ smartio_tool return 80000

Running smartio_example SISCI demo program:

$ smartio_example -dev 80c00
Shared mem[0]:
0x00000000
BAR[0]:
0x0e7330a2
IO address of local segment seen from the device 80c00: 410e1000
IO address of remote shared device memory seen from the device 80c00: 48b8e1000

Exit code¶

Exits with standard errno values. Some value have special meaning, see table below. There can sometimes be useful error messages in the kernel log (dmesg) after a command fails.

Value	ERRNO	Meaning
EIO	5	Unable to communicate with other node
EBUSY	16	Device is busy or not available
ENODEV	19	Invalid device specified
ENOMEM	12	Failed to allocate a buffer, or to map BARs or DMA window.