Multi-host sharing of an NVMe drive using dis_nvme

Dolphin has developed a prototype NVMe driver that allows multiple nodes to access shared NVMe devices at the same time, without requiring that the devices support SR-IOV. This is made possible by the highly parallel nature of the NVMe device-driver model. Using the dis_nvme driver exposes the same block device on all cluster nodes. This enables use of shared-disk filesystems like GFS2 and OCFS2.

Disclaimer

The dis_nvme driver is currently only provided as a development version and not yet a finalized product. Dolphin does not provide any guarantees with regards to functionality or data integrity.

Requirements

Additional System Considerations

  • The NTB prefetchable size on the lender limits the amount of RAM the NVMe drive can access at any given moment. A small prefetchable size can negatively impact performance.

eXpressWare and dis_nvme Installation

When installing eXpressWare, make sure to request installation of SmartIO, either interactively or by passing the --enable-smartio argument. Please refer to the installation guide for more details. After eXpressWare installation, install dis_nvme following instructions from Dolphin.

Lending NVMe drives to the Pool

The devices that are going to be shared must be added and made available with smartio_tool add and smartio_tool available. The lender must also be connected to all the borrowers with smartio_tool connect. See Lending Local Devices for more details.

Loading and using dis_nvme

Load the dis_nvme manager on one of the cluster nodes. On all nodes that will need access to the drive, load the dis_nvme client module. The modules will automatically discover all available NVMe drives and all nodes will get a new /dev/disnvme### for each available NVMe drive. At this point you can use the NVMe drives as any other shared disk. Note that dis_nvme only provides shared access to the drives and that a shared-disk filesystem must be used to allow shared access to a single filesystem. Refer to Using Dolphin’s dis_nvme driver for further details.