Glossary

4G decoding

A BIOS option that enables use of 64-bit addresses for BARs. This is typically required for large BARs as the 32-bit address space is limited.

Adapter
adapter id
Host adapter

A supported PCI Express cable adapter. This is the PCI Express hardware installed in the Cluster Nodes. Some Cluster Nodes may have an integrated PCIe Express chipset that connects the system to a PCIe enabled backplane. For the sake of this guide, we still in some cases refer to this as an adapter.

Adapter number

Each adapter in a Cluster Node is identified by an adapter number.

Each PCIe network connections in the SBC is identified by an adapter number . Most SBCs only support one connection

BAR
BARs

PCIe devices exposes registers and memory in ranges referred to as BARs. A PCIe device can have from zero to 6 BARs. The BARs a device exposes can be viewed with lspci.

BDF
bdf
device-function
devfn

A BDF or bus-device-function specifies a pci device in a system. It’s specified as a series of hexadecimal integers delimited as follows: domain:bus:device.function. Running lspci reveals the BDF of each device in the system. Example: 01:00.0

Cluster

All Cluster Nodes constitute the cluster.

Cluster Management Node (frontend)

The single computer that is running software that monitors and controls the nodes in the cluster. For increased fault tolerance, the Cluster Management Node should ideally not be part of the PCI Express interconnect it controls. Instead, the Cluster Management Node should communicate with the Cluster Nodes out-of-band, which means via Ethernet.

The single computer that is running software that monitors and configures the Cluster Nodes. The light weight cluster management service communicates with the Cluster Nodes out-of-band, which means via Ethernet.

The Cluster Management functionality is not supported or required for systems running VxWorks.

The Cluster Management functionality is not supported or required for systems running RTX.

CPU architecture

The CPU architecture relevant in this guide is characterized by the addressing width of the CPU (32 or 64 bit) and the instruction set (x86, PowerPC, Sparc, ARM etc.). If these two characteristics are identical, the CPU architecture is identical for the scope of this guide.

DIP
dip-setting
dip switch

A small array of switches, typically found on a PCB. See wikipedia article

DX

This is a PCI Express Gen1 compliant interconnect based on the ASI standard. It requires a dedicated switch to build clusters larger than 2 Cluster Nodes. The DX product family was introduced on the market in 2006. It consists of a PCI Express Adapter card (DXH510), a 10 port switch (DXS510) and 8 slot PCI Expansion box (DXE510).

EEPROM
Firmware

EEPROM refers to the configuration and microcode running on an adapter card or a switch.

Fabric

A fabric is an independent, closed communication network that connects a number of machines (here: all nodes in your cluster). Thus, with one adapter in each Cluster Node and all PCIe connections set up, the cluster is using a single fabric. Adding another adapter to each Cluster Node and connecting them would make a cluster with 2 fabrics.

Installation machine

The installation script is typically executed on the Cluster Management Node, but can also be executed on another machine that is neither a Cluster Node nor the Cluster Management Node, but has network (ssh) access to all Cluster Nodes and the Cluster Management Node. This machine is the installation machine.

INX

This is a PCI Express Gen 3 interconnect from Dolphin based on the Intel NTB functionality available with some Intel CPUs. The software was introduced in July 2014.

IOMMU
iommu
vt-d

An IOMMU (IO Memory Mapping Unit) enables a CPU to create virtual address spaces for IO devices in the same way the MMU does for the CPU. This can be used to isolate devices from each other. Typically used for virtualization.

IX

This is a PCI Express Gen 2 interconnect from Dolphin based on standard PCI Express chips from IDT. The firs PCI Express Adapter card (IXH610) was introduced on the market in December 2010. A compliant XMC Adapter (IXH620), 7 slot Expansion box (IXE600) and a 8 port switch box (IXS600) was added to the interconnect family in 2011.

The IX keyword is used for specifying a driver that supports PCI Express Gen2 chips from IDT.

Kernel build machine

The interconnect drivers are kernel modules and thus need to be built for the exact kernel running on the node (otherwise, the kernel will refuse to load them). To build kernel modules on a machine, the kernel-specific include files and kernel configuration have to be installed - these are not installed by default on most distributions. You will need to have one kernel build machine available which has these files installed (contained in the kernel-devel RPM that matches the installed kernel version) and that runs the exact same kernel version as the Cluster Nodes. Typically, the kernel build machine is one of the Cluster Nodes itself, but you can choose to build the kernel modules on any other machine that fulfills the requirements listed above.

The cable between two adapters or between an adapter and a PCIe switch.

The PCIe link between two SBCs or between a SBC and a PCIe switch card.

MSI
MSI-X
Message Signaled Interrupts

MSI is the mechanism used by PCIe devices to deliver interrupts to the CPU. To be able to deliver multiple distinct interrupts, some devices support MSI-X (example: network adapters and NVMe drivers).

MXH
MX
MXS

This is a PCI Express Gen 3 interconnect from Dolphin based on standard PCI Express chips from Microsemi. The firs PCI Express Adapter card (MXH830) was introduced on the market in September 2017.

Network Manager

The Dolphin Network Manager is a daemon process named dis_networkmgr running on the Cluster Management Node. It is part of the Dolphin software stack and manages and controls the cluster using the Node Manager running on all Cluster Nodes. The Network Manager knows the interconnect status of all Cluster Nodes.

The service name of the Network Manager is dis_networkmgr.

Node
Cluster Nodes
Cluster Node

A computer which is part of the PCI Express interconnect, which means it has a PCI Express network connection to other nodes. All Cluster Nodes together constitute the cluster. A Cluster Node can be attached to multiple independent fabrics.

Node Manager

The Node Manager is a daemon process that is running on each Cluster Node and provides remote access to the interconnect driver and other Cluster Node status to the Network Manager. It reports status and performs actions like configuring the installed adapter.

The service name of the Node Manager is dis_nodemgr.

The Node Manager and Network Manager is not required for VxWorks systems.

NodeId
node id

Each Cluster Node in a fabric is identified by an assigned NodeId. Similar concept to an IP address.

NTB
Non-Transparent
Non-Transparent Bridge

An NTB is a PCIe device used to interconnect separate PCIe trees

NVMe
nvme

Non-volatile memory express is a PCIe-based SSD drive.

p2p
P2P
peer-to-peer

Peer to peer support refers to the ability of PCIe devices to send transactions to eachother. This is not supported on all CPUs / motherboards.

prefetchable memory
BAR2
prefetch

Prefetchable memory size refers to the size the window (typically BAR2) that an NTB can map remote memory through.

PXH
PX

This is a PCI Express Gen 3 interconnect from Dolphin based on standard PCI Express chips from Broadcom/Avago/PLX. The first PCI Express Adapter card (PXH810) was introduced on the market in October 2015. The PXH810 and PXH812 cards are complaint with the IXS600 8 port switch.

The PCI Express Gen3 x16 MiniSAS-HD based PXH830 card was introduced in 2016.

The PX keyword is used for specifying a driver that supports PCI Express Gen2 and Gen3 chips from Broadcom/Avago/PLX.

SBC

Single Board Computer (SBC).

SCI / D

D is the abbreviation for the Scalable Coherent Interface (SCI) class of host adapters and Dolphin’s SCI interconnect introduced in 1993. SCI uses a switchless topology and therefore scales easily to large Clusters.

self-installing archive (SIA)

A self-installing archive (SIA) is a single executable shell command file (for Linux and Solaris) that is used to compile and install the Dolphin software stack in all required variants. It largely simplifies the deployment and management of a PCI Express based cluster.

SISCI

SISCI (Software Infrastructure for Shared-Memory Cluster Interconnect) is the user-level API to create applications that make direct use of the low level PCI Express interconnect shared memory capabilities.

To run SISCI applications, a service named dis_sisci has to be running; it loads the required kernel module and sets up the SISCI devices.

SuperSockets

SuperSockets is a Berkeley sockets compliant socket API provided by Dolphin. SuperSockets is currently supported on systems using Linux and Windows.

Target Adapter

Adapter card used in the upstream slot of an expansion box. For example MXH832 and PXH832.

Transparent

The normal use case for PCIe is to connect periphiral devices. To differentiate from the Non-Transparent usage this is refered to as “transparent”. PCIe is the ‘normal’ usage of PCIe.

VxWorks Windows Installer (MSI)

The VxWorks Windows Installer is an engine for the installation, maintenance, and removal of the VxWorks software on modern Microsoft Windows systems. The installation information, and often the files themselves, are packaged in installation packages, loosely relational databases structured as OLE Structured Storage Files and commonly known as “MSI files”, from their default file extension.

Windows Installer (MSI)

The Windows Installer is an engine for the installation, maintenance, and removal of software on modern Microsoft Windows systems. The installation information, and often the files themselves, are packaged in installation packages, loosely relational databases structured as OLE Structured Storage Files and commonly known as “MSI files”, from their default file extension.

write-combining

Write-Combining is a CPU feature designed to increase write bandwidth to an uncached memory region. It allows writes to be merged, reordered or speculatively delayed.

x1, x2, x4, x8, x16

PCI express combine multiple lanes (serial high-speed communication channels using few electrical connections) into communication paths with a higher bandwidth. With PCI Express Gen. 1, each lane carries 2.5Gbit/s of traffic, with PCI Express Gen 2, each lane carries 5.0 Gbit/s and with PCI Express Gen3, each lane carries 8.0 Gibt/s. Combining 8 lanes into a single communication path is called x8 and thus delivers 40Gbit/s Bandwidth for Gen 2 or 64Bbit/s Bandwidth for Gen 3, while x16 doubles this bandwidth using 16 lanes and delivers 128Gbit/s for Gen3 in each direction.