Windows Hot-Add

Windows Hot-Add is a feature of eXpressWare that enables transparent I/O expansion while allowing devices to be added or removed without shutting down the host system. This includes connecting or disconnecting a cable to the expansion box, powering off the expansion box, adding or replacing a device, and then powering it back on.

We use proprietary techniques to reserve PCIe resources from Windows, allowing you to add more devices than were present at boot time. You must therefore select a Dolphin NTB card configuration that reserves sufficient resources.

Prerequisites

  • A Windows host machine with eXpressWare in the Transparent Board Management mode. Please consult the installation manual for instructions.

  • A supported Dolphin NTB card (e.g. Dolphin MXH530) that will be used in the host machine. Please consult the product users guide or firmware release note for details about Windows Hot-Add support.

  • A Dolphin NTB or Transparent target card that will be used in the expansion (e.g. Dolphin MXH530).

  • A passive expansion backplane (e.g. Dolphin IBP-G5X16-2) or an active expansion box (e.g. Dolpin eBox).

  • An understanding of how much prefetchable BAR space your devices are going to use.

Configuring

  1. Set the DIP settings of the Dolphin NTB card into the correct position for the Windows Hot-Add configuration. Some Dolphin NTB cards support various configurations for downstream port bifurcation and different amounts of prefetchable BAR space reservations. Please consult the product users guide for details.

  2. Insert the NTB card into the host machine. Do not plug any cables yet.

  3. Boot and sign in to the machine.

  4. Power off the machine to complete the configuration.

Now, you are ready to use Windows Hot-Add. You can now plug or unplug the cable to the target card at any time. The same applies to adding and removing new devices, or powering on or off the expansion box.

Using dis_diag

When a Hot-Add config is detected at the host, dis_diag will provide additional information about the adapter:

Hot-Add config            : Yes
Hot-Add bound DSP         : Yes
Hot-Add add count         : 0
Hot-Add remove count      : 0
Hot-Add reserved buses    : 241
Hot-Add reserved nonpref  : 64.00 MB
Hot-Add reserved pref     : 256.00 MB

The Hot-Add config will always show, with either Yes or No. The rest of the lines are only visible if the Hot-Add config is actually detected. The next line, Hot-Add bound DSP, tells whether the downstream port(s) has been configured correctly. The next two lines will increment when downstream devices are added and removed. The final three lines describe the amount of PCIe bus numbers, non-prefetchable and prefetchable BAR space that was reserved for downstream devices.

Limitations

Bus number reservation

Our ability to obtain additional PCIe resources is constrained by the system’s enumeration process, which is outside our control. As a result, it is difficult to determine in advance how many PCIe bus numbers will be available for downstream devices.

The number of available PCIe bus numbers depends on the placement of other devices in the system. For example, when the Dolphin NTB host card is assigned a lower bus number and no higher-numbered devices are present, the system can often support well over 240 downstream bus numbers, enabling large-scale device connectivity.

In configurations where higher-numbered devices are already present, the available range may be smaller after internal requirements are accounted for. This can be the case of dual socket systems where the Dolphin NTB is placed near the top of the first socket’s bus number space, with the next socket nearby. This limitation can often be avoided by installing the card in a higher-numbered PCIe slot on the last socket, which maximizes the available bus number range for downstream devices.