Glossary

Adapter
adapter id
Host adapter

A supported PCI Express cable adapter. This is the PCI Express hardware installed in the Cluster Nodes. Some Cluster Nodes may have an integrated PCIe Express chipset that connects the system to a PCIe enabled backplane. For the sake of this guide, we still in some cases refer to this as an adapter.

prefetchable memory
BAR2
prefetch

Prefetchable memory size refers to the size the window (typically BAR2) that an NTB can map remote memory through.

BAR
BARs

PCIe devices exposes registers and memory in ranges referred to as BARs. A PCIe device can have from zero to 6 BARs. The BARs a device exposes can be viewed with lspci.

BDF
bdf
device-function
devfn

A BDF or bus-device-function specifies a pci device in a system. It’s specified as a series of hexadecimal integers delimited as follows: domain:bus:device.function. Running lspci reveals the BDF of each device in the system. Example: 01:00.0

NTB
Non-Transparent
Non-Transparent Bridge

An NTB is a PCIe device used to interconnect separate PCIe trees

Transparent

The normal use case for PCIe is to connect peripheral devices. To differentiate from the Non-Transparent usage this is referred to as “transparent”. PCIe is the ‘normal’ usage of PCIe.

Target Adapter

Adapter card used in the upstream slot of an expansion box. For example MXH832 and PXH832.

Adapter number

Each adapter in a Cluster Node is identified by an adapter number.

Node
Cluster Nodes
Cluster Node
cluster node
cluster nodes

A computer which is part of the PCI Express interconnect, which means it has a PCI Express network connection to other nodes. All cluster nodes together constitute the cluster. A cluster node can be attached to multiple independent fabrics.

SBC

Single Board Computer (SBC).

Fabric

A fabric is an independent, closed communication network that connects a number of machines (here: all nodes in your cluster). Thus, with one adapter in each Cluster Node and all PCIe connections set up, the cluster is using a single fabric. Adding another adapter to each Cluster Node and connecting them would make a cluster with 2 fabrics. A Host may be connected by several parallel Fabrics.

The cable between two adapters or between an adapter and a PCIe switch.

The PCIe link between two SBCs or between a SBC and a PCIe switch card.

Cluster

All Cluster Nodes constitute the cluster.

IX

This is a PCI Express Gen 2 interconnect from Dolphin based on standard PCI Express chips from IDT. The firs PCI Express Adapter card (IXH610) was introduced on the market in December 2010. A compliant XMC Adapter (IXH620), 7 slot Expansion box (IXE600) and a 8 port switch box (IXS600) was added to the interconnect family in 2011.

The IX keyword is used for specifying a driver that supports PCI Express Gen2 chips from IDT.

PXH
PX

This is a PCI Express Gen 3 interconnect from Dolphin based on standard PCI Express chips from Broadcom/Avago/PLX. The first PCI Express Adapter card (PXH810) was introduced on the market in October 2015. The PXH810 and PXH812 cards are complaint with the IXS600 8 port switch.

The PCI Express Gen3 x16 MiniSAS-HD based PXH830 card was introduced in 2016.

The PX keyword is used for specifying a driver that supports PCI Express Gen2 and Gen3 chips from Broadcom/Avago/PLX.

MXH
MX
MXS

This is a PCI Express Gen 3 interconnect from Dolphin based on standard PCI Express chips from Microsemi. The first PCI Express Adapter card (MXH830) was introduced on the market in September 2017.

SuperSockets

SuperSockets is a Berkeley sockets compliant socket API provided by Dolphin. SuperSockets is currently supported on systems using Linux and Windows.

SISCI

SISCI (Software Infrastructure for Shared-Memory Cluster Interconnect) is the user-level API to create applications that make direct use of the low level PCI Express interconnect shared memory capabilities.

To run SISCI applications, a service named dis_sisci has to be running; it loads the required kernel module and sets up the SISCI devices.

NodeId
node id

Each Cluster Node in a fabric is identified by an assigned NodeId. Similar concept to an IP address.

x1, x2, x4, x8, x16

PCI express combine multiple lanes (serial high-speed communication channels using few electrical connections) into communication paths with a higher bandwidth. With PCI Express Gen. 1, each lane carries 2.5Gbit/s of traffic, with PCI Express Gen 2, each lane carries 5.0 Gbit/s and with PCI Express Gen3, each lane carries 8.0 Gbit/s. Combining 8 lanes into a single communication path is called x8 and thus delivers 40Gbit/s Bandwidth for Gen 2 or 64Gbit/s Bandwidth for Gen 3, while x16 doubles this bandwidth using 16 lanes and delivers 128Gbit/s for Gen3 in each direction.

4G decoding

A BIOS option that enables use of 64-bit addresses for BARs. This is typically required for large BARs as the 32-bit address space is limited.

p2p
P2P
peer-to-peer
Peer to Peer communication

Peer-to-peer support refers to the ability of PCIe devices to send transactions to each other without involving the system memory. This is not supported on all CPUs / motherboards.

IOMMU
iommu
vt-d

An IOMMU (IO Memory Mapping Unit) enables a CPU to create virtual address spaces for IO devices in the same way the MMU does for the CPU. This can be used to isolate devices from each other. Typically used for virtualization.

DIP
dip-setting
dip switch

A small array of switches, typically found on a PCB. See wikipedia article <https://en.wikipedia.org/wiki/DIP_switch>_

EEPROM
Firmware

EEPROM refers to the configuration and microcode running on an adapter card or a switch.

write-combining

Write-Combining is a CPU feature designed to increase write bandwidth to an uncached memory region. It allows writes to be merged, reordered or speculatively delayed.

PCI
PCIe

PCI Express (Peripheral Component Interconnect Express) is a widely used high-speed interconnect mostly used to connect devices such as GPUs, Network Cards and flash storage to a system.

root complex
Root Complex

The device that connects the CPU and memory to the PCIe fabric. Often synonymous with CPU.