Writing drivers with SmartIO

This chapter shows how SISCI can be utilized to write a user space device driver. Such a device driver built using SISCI can take full advantage of all SISCI features and functionalities. This part of the SISCI API builds on top of SmartIO, Dolphin’s device sharing technology. Devices are only usable with SISCI after they’ve been added and made available with SmartIO. Please refer to Adding Devices to the Pool for more details.

Finding and borrowing the device

A SmartIO device is designated by it’s fabric device id (fdid). The fdid for a device may be discovered externaly using smartio_tool or dynamically in the application itself. A list of devices may be acquired using SCIGetDeviceList. Optionally a sci_smartio_device_info_t filter may be provided to SCIGetDeviceList to only list devices with the given properties. A device driver would normally provide a filter that only matches compatible devices. SCIGetDeviceList may be called multiple times with different filters.

sci_smartio_device_info_t filter = {0};

filter.pci_class = 0x010802; /* Filter on NVMe drives */

/* Get a list of discovered SmartIO devices matching filter */
num_devices = SCIGetDeviceList(sd, device_ids, 64, &filter, 0, &scierr);

/* Loop through discovered devices and try to borrow one of them */
for (device_it = 0; device_it < num_devices; ++device_it) {

  SCIBorrowDevice(sd, &device->device, device_ids[device_it], 
            SCI_FLAG_EXCLUSIVE, &scierr);
    if (scierr == SCI_ERR_OK) {
        break;
    }
}

Before a device can be used it must be borrowed This is accomplished with SCIBorrowDevice. If this function succeeds, it will provide a sci_smartio_device_t that must be used in most other SmartIO SISCI API functions. A device may Optionally be acquired exclusively using SCI_FLAG_EXCLUSIVE. When this flag is used SmartIO guarantees that the caller is the only user of the device. If any other user has borrowed the device, SCIBorrowDevice with SCI_FLAG_EXCLUSIVE will fail.

To release use of a device SCIReturnDevice must be called. This invalidates the sci_smartio_device_t handle.

Mapping device BARs

The BAR / device registers can be mapped like a remote segment. Each of the device’s BARs are represented as a remote segment. These segments must be connected to by using SCIConnectDeviceSegment. After this, the segment can be used like any other remote segment. In other words you need to call SCIMapRemoteSegment to map the registers into your process and get a virtual address.

Note

The default behavior of SCIMapRemoteSegment is to use write-combining. This may cause undesired effects when mapping device registers. To control the ordering and to avoid merging of writes, SCIFlush must be used. Alternativly, write-combining can be disabled for a mapping by passing the flag SCI_FLAG_IO_MAP_IOSPACE. Note that this will reduce the performance for bulk data transfers for this mapping. You should use different mappings for bulk data transfers and register accesses if possible.

SCIConnectDeviceSegment(device, &segment, id, type, NULL, NULL, 0, &scierr);
if (scierr != SCI_ERR_OK) {
    return scierr;
}

size = SCIGetRemoteSegmentSize(segment);

bar_ptr = SCIMapRemoteSegment(segment, &map_handle, 0, size, NULL,
     SCI_MEMTYPE_BAR, &err);

Setting up DMA mappings

The device may be given access to DMA into any local or remote segment. To map a local segment use SCIMapLocalSegmentForDevice. This function provides the caller with a remoteAddr. This is the bus address to be used by your device in order to reach the given segment.

Mapping remote segments is done using SCIMapRemoteSegmentForDevice. Note that this function takes a sci_remote_segment_t. This means that you must first connect to the desired remote segment using SCIConnectSegment

SCIMapRemoteSegmentForDevice(segment, device, &ioaddr, 0, size, 0, &scierr);
if (scierr != SCI_ERR_OK) {
   return scierr;
}

/* Tell the device what address to use */
*device_dma_register = ioaddr;

It’s also possible to map BARs of other devices and allow a device to perform P2P transfers to the other device. In this case you should first connect to the other device’s bar using SCIConnectDeviceSegment and then pass the sci_remote_segment_t to SCIMapRemoteSegmentForDevice.

Examples

Examples can be found in the sisci-smartio-examples repository on GitHub