Glossary

4G decoding

A BIOS option that enables use of 64-bit addresses for BARs. This is typically required for large BARs as the 32-bit address space is limited.

Adapter
adapter id
Host adapter

A supported PCI Express cable adapter. This is the PCI Express hardware installed in the Cluster Nodes. Some Cluster Nodes may have an integrated PCIe Express chipset that connects the system to a PCIe enabled backplane. For the sake of this guide, we still in some cases refer to this as an adapter.

Adapter number

Each adapter in a Cluster Node is identified by an adapter number.

Each PCIe network connections in the SBC is identified by an adapter number . Most SBCs only support one connection

BAR
BARs

PCIe devices exposes registers and memory in ranges referred to as BARs. A PCIe device can have from zero to 6 BARs. The BARs a device exposes can be viewed with lspci.

BDF
bdf
device-function
devfn

A BDF or bus-device-function specifies a pci device in a system. It’s specified as a series of hexadecimal integers delimited as follows: domain:bus:device.function. Running lspci reveals the BDF of each device in the system. Example: 01:00.0

borrow
borrowed

In SmartIO, device are borrowed to gain temporary access to the device.

borrower
lender
lending side
borrow side

In SmartIO, the borrower is the side that borrows devices from the lender side. The lender in turn lends out devices to “borrowers”. A node may be both a lender and a borrower. Similar to a client and server concept.

Cluster

All Cluster Nodes constitute the cluster.

device lending

Device lending is a feature of SmartIO that enables remote devices to be assigned to a system as a local PCIe device.

DIP
dip-setting
dip switch

A small array of switches, typically found on a PCB. See wikipedia article <https://en.wikipedia.org/wiki/DIP_switch>_

EEPROM
Firmware

EEPROM refers to the configuration and microcode running on an adapter card or a switch.

Fabric

A fabric is an independent, closed communication network that connects a number of machines (here: all nodes in your cluster). Thus, with one adapter in each Cluster Node and all PCIe connections set up, the cluster is using a single fabric. Adding another adapter to each Cluster Node and connecting them would make a cluster with 2 fabrics.

fabric device id
fdid

id specifying a single SmartIO device in a fabric.

IOMMU
iommu
vt-d

An IOMMU (IO Memory Mapping Unit) enables a CPU to create virtual address spaces for IO devices in the same way the MMU does for the CPU. This can be used to isolate devices from each other. Typically used for virtualization.

IX

This is a PCI Express Gen 2 interconnect from Dolphin based on standard PCI Express chips from IDT. The firs PCI Express Adapter card (IXH610) was introduced on the market in December 2010. A compliant XMC Adapter (IXH620), 7 slot Expansion box (IXE600) and a 8 port switch box (IXS600) was added to the interconnect family in 2011.

The IX keyword is used for specifying a driver that supports PCI Express Gen2 chips from IDT.

The cable between two adapters or between an adapter and a PCIe switch.

The PCIe link between two SBCs or between a SBC and a PCIe switch card.

MSI
MSI-X
Message Signaled Interrupts

MSI is the mechanism used by PCIe devices to deliver interrupts to the CPU. To be able to deliver multiple distinct interrupts, some devices support MSI-X (example: network adapters and NVMe drivers).

MXH
MX
MXS

This is a PCI Express Gen 3 interconnect from Dolphin based on standard PCI Express chips from Microsemi. The first PCI Express Adapter card (MXH830) was introduced on the market in September 2017.

Node
Cluster Nodes
Cluster Node
cluster node
cluster nodes

A computer which is part of the PCI Express interconnect, which means it has a PCI Express network connection to other nodes. All cluster nodes together constitute the cluster. A cluster node can be attached to multiple independent fabrics.

NodeId
node id

Each Cluster Node in a fabric is identified by an assigned NodeId. Similar concept to an IP address.

NTB
Non-Transparent
Non-Transparent Bridge

An NTB is a PCIe device used to interconnect separate PCIe trees

NVMe
nvme

Non-volatile memory express is a PCIe-based SSD drive.

p2p
P2P
peer-to-peer

Peer-to-peer support refers to the ability of PCIe devices to send transactions to each other. This is not supported on all CPUs / motherboards.

PCI
PCIe

PCI Express (Peripheral Component Interconnect Express) is a widely used high-speed interconnect mostly used to connect devices such as GPUs, Network Cards and flash storage to a system.

prefetchable memory
BAR2
prefetch

Prefetchable memory size refers to the size the window (typically BAR2) that an NTB can map remote memory through.

PXH
PX

This is a PCI Express Gen 3 interconnect from Dolphin based on standard PCI Express chips from Broadcom/Avago/PLX. The first PCI Express Adapter card (PXH810) was introduced on the market in October 2015. The PXH810 and PXH812 cards are complaint with the IXS600 8 port switch.

The PCI Express Gen3 x16 MiniSAS-HD based PXH830 card was introduced in 2016.

The PX keyword is used for specifying a driver that supports PCI Express Gen2 and Gen3 chips from Broadcom/Avago/PLX.

root complex
Root Complex

The device that connects the CPU and memory to the PCIe fabric. Often synonymous with CPU.

SBC

Single Board Computer (SBC).

SISCI

SISCI (Software Infrastructure for Shared-Memory Cluster Interconnect) is the user-level API to create applications that make direct use of the low level PCI Express interconnect shared memory capabilities.

To run SISCI applications, a service named dis_sisci has to be running; it loads the required kernel module and sets up the SISCI devices.

SmartIO
sio
dis_sio

Dolphin SmartIO refers to the SmartIO component of eXpressWare

SuperSockets

SuperSockets is a Berkeley sockets compliant socket API provided by Dolphin. SuperSockets is currently supported on systems using Linux and Windows.

Target Adapter

Adapter card used in the upstream slot of an expansion box. For example MXH832 and PXH832.

target device
target devices
target device driver

In SmartIO, the target device refers to the device that is borrowed. For device lending target device driver refers to the device driver that binds to a borrowed, remote device.

Transparent

The normal use case for PCIe is to connect peripheral devices. To differentiate from the Non-Transparent usage this is referred to as “transparent”. PCIe is the ‘normal’ usage of PCIe.

write-combining

Write-Combining is a CPU feature designed to increase write bandwidth to an uncached memory region. It allows writes to be merged, reordered or speculatively delayed.

x1, x2, x4, x8, x16

PCI express combine multiple lanes (serial high-speed communication channels using few electrical connections) into communication paths with a higher bandwidth. With PCI Express Gen. 1, each lane carries 2.5Gbit/s of traffic, with PCI Express Gen 2, each lane carries 5.0 Gbit/s and with PCI Express Gen3, each lane carries 8.0 Gbit/s. Combining 8 lanes into a single communication path is called x8 and thus delivers 40Gbit/s Bandwidth for Gen 2 or 64Gbit/s Bandwidth for Gen 3, while x16 doubles this bandwidth using 16 lanes and delivers 128Gbit/s for Gen3 in each direction.