Privileged functions

The SISCI API contains some functions that may cause severe system problems; corrupting data or causing crashes if used incorrectly.

Registering physical memory as a segment

The SISCI API also allows physical memory, e.g. memory on a PCIe device (FPGA, GPU etc.), to be registered as a SISCI segment. The requirements are that the user knows the physical address and its size, and that the memory is always available (main memory needs to be locked down).

If registering a PCI/PCIe device, the user also needs to ensure the local system supports direct PCI Express peer to peer transfers (the network cards are able to directly access the physical memory device).

The concept is to allocate an “empty” segment – without any memory allocated. This is done by specifying the flag SCI_FLAG_EMPTY when calling SCICreateSegment() and later on, use the SCIAttachPhysicalMemory() function to register the physical address.

The following code will demonstrate how to do this:

sci_error_t error;
sci_local_segment_t segment;
sci_ioaddr_t ioa = 0x1234567800000000; /* 64 bit physical address of memory */

SCICreateSegment(... , SCI_FLAG_EMPTY, ...); /* initialization */
If (error == SCI_ERR_OK) {

    SCIAttachPhysicalMemory(ioa, NULL, 0, size, segment, 0, &error);
    if(error != SCI_ERR_OK) {
        /* Terminate or handle error */
    }
}

/* from this point we can use it as a regular segment */

Once the physical segment is properly registered and set up, it is managed and used just as a regular SISCI segment. A full example of how to work with physical segments can be found in the SISCI source code example rpcia.c

Querying the remote address of a mapped SISCI segment

If you would like to set up a local FPGA, GPU, etc. to access a remote SISCI segment directly, you need to identify its local physical address. This address will typically reside within the network adapter’s address space and will only be valid as long as the remote segment is and remains mapped from this system. The following code can be used to identify this address:

query.subcommand = SCI_Q_REMOTE_SEGMENT_IOADDR;
query.segment = segment;

SCIQuery(SCI_Q_REMOTE_SEGMENT, &query, 0, &error);
if(error == SCI_ERR_OK) {
    printf("Remote segment ioaddr = 0x%0*llXn", 16, query.data.ioaddr);
}

It is up to the programmer to find a good way to propagate this address down to the device that wants to use this address.