Writing drivers with SmartIO SISCI API

The SISCI SmartIO functions provides an API for low level access to PCIe devices in the cluster. It can be used to write user space drivers that operate on devices located anywhere in the cluster. Unlike, device lending, multiple users can ‘borrow’ the same device at the same time. This allows distributed device drivers to be created. The SmartIO API is compatible with the rest of the SISCI API so that the ‘normal’ SISCI functions can be used when writing device drivers, for instance this allows the driver instances to communicate with each other. The DMA engine on the NTB adapter can also be used to move data to and from a SmartIO device.

Examples can be found in the sisci-smartio-examples repository on GitHub

Finding and borrowing the device

A SmartIO device is designated by it’s fabric device id (fdid). The fdid for a device may be discovered externaly using smartio_tool or dynamically in the application itself. A list of devices may be acquired using SCIGetDeviceList(). Optionally a sci_smartio_device_info_t filter may be provided to SCIGetDeviceList() to only list devices with the given properties. A device driver would normally provide a filter that only matches compatible devices. SCIGetDeviceList() may be called multiple times with different filters.

 sci_smartio_device_info_t filter = {0};

 filter.pci_class = 0x010802; /* Filter on NVMe drives */

 /* Get a list of discovered SmartIO devices matching filter */
 num_devices = SCIGetDeviceList(sd, device_ids, 64, &filter, 0, &scierr);

 /* Loop through discovered devices and try to borrow one of them */
 for (device_it = 0; device_it < num_devices; ++device_it) {

   SCIBorrowDevice(sd, &device->device, device_ids[device_it],
             SCI_FLAG_EXCLUSIVE, &scierr);
     if (scierr == SCI_ERR_OK) {
         break;
     }
}

Before a device can be used it must be borrowed. This is accomplished with SCIBorrowDevice(). If this function succeedes, it will provide a sci_smartio_device_t that must be used in most other SmartIO SISCI API functions. A device may Optionally be acquired exclusivly using SCI_FLAG_EXCLUSIVE. When this flag is used SmartIO guarantees that the caller is the only user of the device. If any other user has borrowed the device, SCIBorrowDevice() with SCI_FLAG_EXCLUSIVE will fail.

To release use of a device SCIReturnDevice() must be called. This invalidates the sci_smartio_device_t handle.

Mapping device BARs

The BAR / device registers can be mapped like a remote segment. Each of the device’s BARs are represented as a remote segment. These segments must be connected to by using SCIConnectDeviceSegment(). After this, the segment can be used like any other remote segment. In other words you need to call SCIMapRemoteSegment() to map the registers into your process and get a virtual address.

Note

The default behavior of SCIMapRemoteSegment() is to use write-combining. This may cause undesired effects when mapping device registers. To control the ordering and to avoid merging of writes, SCIFlush() must be used. Alternativly, write-combining can be disabled for a mapping by passing the flag SCI_FLAG_IO_MAP_IOSPACE. Note that this will reduce the performance for bulk data transfers for this mapping. You should use different mappings for bulk data transfers and register accesses if possible.

SCIConnectDeviceSegment(device, &segment, id, type, NULL, NULL, 0, &scierr);
if (scierr != SCI_ERR_OK) {
    return scierr;
}

size = SCIGetRemoteSegmentSize(segment);

bar_ptr = SCIMapRemoteSegment(segment, &map_handle, 0, size, NULL,
     SCI_MEMTYPE_BAR, &err);

Setting up DMA mappings

The device may be given access to DMA into any local or remote segment. To map a local segment use SCIMapLocalSegmentForDevice(). This function provides the caller with a remoteAddr. This is the bus address to be used by your device in order to reach the given segment.

Mapping remote segments is done using SCIMapRemoteSegmentForDevice(). Note that this function takes a sci_remote_segment_t. This means that you must first connect to the desired remote segment using SCIConnectSegment().

SCIMapRemoteSegmentForDevice(segment, device, &ioaddr, 0, size, 0, &scierr);
if (scierr != SCI_ERR_OK) {
   return scierr;
}

/* Tell the device what address to use */
*device_dma_register = ioaddr;

It’s also possible to map BARs of other devices and allow a device to perform P2P transfers to the other device. In this case you should first connect to the other device’s bar using SCIConnectDeviceSegment() and then pass the sci_remote_segment_t to SCIMapRemoteSegmentForDevice().