Adapter Prefetchable Memory Size

The prefetchable memory region of an NTB adapter is used for mapping remote segments. When a remote segment with size 32MiB is mapped, 32MiB of the adapter’s prefetch space is consumed util the segment is unmapped. The factory default prefetchable memory size of the Dolphin adapters is 256MiB. Due to various hardware constrains such as alignment requirements the full prefetchable memory space might not be usable. The prefetchable memory space size can be increased if needed. The currently configured prefetchable memory size can be queried by running:

[root@PXMXS-01 ~]# dis_config -gpms
Card 1 - Prefetch memory space size: 512

Configuring the Prefetchable Memory Size

If you intend to increase the prefetchable size beyond the factory default, it’s strongly recommended to ensure that 4G decoding is enabled in the system BIOS before proceeding. This is sometimes also referred to as “64-bit BAR support” “64-bit address decoding”. Consult your system or motherboard manual for details. Some systems may fail to boot if the prefetchable memory size is increased beyond the factory default. To recover from this the prefetchable memory size must be decreased.

Hint

In general, systems with 4G decoding enabled will support prefetchable memory of 32GB or more.

Configuring Prefetch Size on IXH and PXH Adapters

The prefetchable memory size is changed using dis_config. The new size is given in MiB. Prefetch size must be a power of two and larger than 1MiB. Example sizes:

• 256MB

• 1024MB

• 32768MB

[root@PXMXS-01 ~]# dis_config -spms 512
Card 1 - Prefetch space memory size is set to 512 MB
A reboot of the machine is required to make the changes take effect.
[root@PXMXS-01 ~]#

If the system has multiple adapters the adapter must be specified using -a N. dis_config may also be used interactivly by invoking it without parameters. In this mode enter the command set-prefetch-mem-size and follow the on-screen instructions.

Configuring Prefetch Size on MXH830 Adapters

MXH Adapters can be configured with either 256MiB (factory default) or 32GiB. This is controlled using a DIP switch on the adapter card. Please refer to the adapter manual for more details. If the system fails to boot after changing a DIP setting, please return the setting to factory default.

• Setting DIP4 to the ON position sets the prefetchable memory size to 32GiB.

• Setting DIP4 to the OFF position sets the prefetchable memory size to 256MiB (factory default).

If the System Fails to Boot

If the system fails to boot after increasing the prefetchable size ensure that the system boots successfully with the card removed. If so, confirm that 4G decoding is enabled in the BIOS. If the system still does not boot with the card installed this likely means that the system does not support prefetchable memory that large. You may want to contact your system vendor. To recover the adapter card you will need to decrease the prefetchable size. This can be done using another system that supports larger prefetchable memory size and then decreasing it using dis_config. Alternativly the card may be reset to factory defaults. Please refer to the adapter manual. This can be found on the Dolphin Support Page.