Shared Memory Segments

The PCI Express technology implements a remote memory access approach that can be used to transfer data between systems and IO devices. An application can map into its own address space a memory segment actually residing on another node; then read and write operations from or to this memory segment are automatically and transparently converted by the hardware in remote operations. This API provides full support for creating and exporting local memory segments, for connecting to and mapping remote memory segments, for checking whether errors have occurred during a data transfer.

The functions included in this category actually concern three different aspects:

Memory management:

• SCICreateSegment()

• SCIRegisterSegmentMemory()

• SCIRemoveSegment()

• SCIPrepareSegment()

Connection management:

• SCISetSegmentAvailable()

• SCISetSegmentUnavailable()

• SCIConnectSegment()

• SCIDisconnectSegment()

Segment events

• SCIWaitForLocalSegmentEvent()

• SCIWaitForRemoteSegmentEvent()

Shared memory operations:

• SCIMapLocalSegment()

• SCIMapRemoteSegment()

• SCIUnmapSegment()

• SCIMemCpy()

• SCICreateMapSequence()

• SCIRemoveSequence()

• SCIStartSequence()

• SCICheckSequence()

• SCIStoreBarrier()

• SCIFlush()

Memory and connection management functions affect the state of a local segment, whose state diagram is shown in the figure below.

        stateDiagram-v2
PREPARED: Prepared (unavailable)
NOT_PREPARED: Created
AVAILABLE: Available

[*] --> NOT_PREPARED: SCICreateSegment
NOT_PREPARED --> PREPARED: SCIPrepareSegment
PREPARED --> AVAILABLE: SCISetSegmentAvailable
AVAILABLE --> PREPARED: SCISetSegmentUnavailable

AVAILABLE --> [*]: SCIRemoveSegment
PREPARED --> [*]: SCIRemoveSegment
NOT_PREPARED --> [*]: SCIRemoveSegment
    

The state of a remote segment, shown in figure below, depends on what happens on the network or on the node where the segment physically resides. The transitions sci_segment_cb_reason_t are marked with callback reasons between the remote segment states.

        stateDiagram-v2

[*] --> CONNECTING: SCIConnectSegment
CONNECTING --> LOST: CB_LOST
OPERATIONAL --> LOST: CB_LOST
NOT_OPERATIONAL --> LOST: CB_LOST
DISCONNECTING_OPERATIONAL --> LOST: CB_LOST
DISCONNECTING_NOT_OPERATIONAL --> LOST: CB_LOST

CONNECTING --> OPERATIONAL: CB_OPERATIONAL
NOT_OPERATIONAL --> OPERATIONAL: CB_OPERATIONAL
DISCONNECTING_NOT_OPERATIONAL --> DISCONNECTING_OPERATIONAL: CB_OPERATIONAL

OPERATIONAL --> NOT_OPERATIONAL: CB_NOT_OPERATIONAL

OPERATIONAL --> DISCONNECTING_OPERATIONAL: CB_DISCONNECT
NOT_OPERATIONAL --> DISCONNECTING_NOT_OPERATIONAL: CB_DISCONNECT

    

State machine for remote segment callbacks

The transitions are marked with callback reasons. SCIDisconnectSegment can be called from each state to exit the state diagram.

Special Functions

Some special functions exists, these are available for all platforms and operating systems, but may only be needed for special environments or purposes. The following function can be used to synchronize the CPU memory cache on systems without coherent IO (Nvidia Jetson Tegra TK1,TX1,TX2).

• SCICacheSync()