Sharing a non-SR-IOV NVMe drive with dis_nvme

Dolphin has developed a NVMe driver from scratch that allows multiple nodes shared access to the same NVMe drive, without requiring SR-IOV support in the NVMe drive. This is made possible by the highly parallel nature of the NVMe device-driver model. Using the dis_nvme driver exposes the same block device on all cluster nodes. This enables use of shared-disk filesystems like GFS2 and OCFS2.

Hint

The Dolphin dis_nvme driver is not yet publically available. Contact support if you’re interrested!

Requirements

Additional System Considerations

  • The NTB prefetchable size on the lender limits the amount of RAM the NVMe drive can access at any given moment. A small prefetchable size can negatively impact performance.

eXpressWare and dis_nvme Installation

When installing eXpressWare, make sure to request installation of SmartIO, either interactively or by passing the --enable-smartio argument. Please refer to the installation guide for more details. After eXpressWare installation, install dis_nvme following instructions from Dolphin.

Lending NVMe drives to the Pool

The devices that are going to be shared must be added and made available with smartio_tool add and smartio_tool available. The lender must also be connected to all the borrowers with smartio_tool connect. See Lending Local Devices for more details.

Loading and using dis_nvme

Load the dis_nvme manager on one of the cluster nodes. On all nodes that will need access to the drive, load the dis_nvme client module. The modules will automatically discover all available NVMe drives and all nodes will get a new /dev/disnvme### for each available NVMe drive. At this point you can use the NVMe drives as any other shared disk. Note that dis_nvme only provides shared access to the drives and that a shared-disk filesystem must be used to allow shared access to a single filesystem.