Accessing memory¶
You can start using a memory segment once it is made available, which means you have a valid handle to either a local or remote segment resource. You can access it in two ways: either you map it into the address space of your process and then you access it as you normally access memory, e.g. via pointer operations, or you can choose the so-called Direct Memory Access (DMA) approach, which delegates the actual data transfer to the network adapter.
In this chapter, you will learn how to use memory-mapped segments, in particular:
How to map a local memory segment into the addressable space of your program.
How to map a remote memory segment into the addressable space of your program.
How to access data within a mapped segment.
The use of DMA is addressed in the DMA chapter.
Mapping a local memory segment into addressable space¶
Let’s start with a local segment:
sci_desc_t v_dev;
sci_local_segment_t local_segment;
sci_map_t local_map;
sci_error_t error;
void* map_address;
SCIInitialize(...);
SCIOpen(&v_dev,...);
SCICreateSegment(v_dev,
&local_segment,
RECEIVER_SEG_ID,
RECEIVER_SEG_SIZE,
NO_CALLBACK,
0,
NO_FLAGS,
&error);
if (error != SCI_ERR_OK) {
sci_error_t rc = error;
SCIClose(m_vConnIntrDev, NO_FLAGS, &error);
SCITerminate();
return error;
}
SCIPrepareSegment(local_segment, ADAPTER_NO, NO_FLAGS, &error);
if (error == SCI_ERR_OK) {
/* the segment has been successfully created */
size_t offset = 0;
size_t size = RECEIVER_SEG_SIZE;
void* suggested_address = 0;
map_address = SCIMapLocalSegment(local_segment,
&local_map,
offset,
size,
suggested_address,
NO_FLAGS,
error);
if (error == SCI_ERR_OK) {
/* the segment has been successfully mapped at virtual
* address map_address */
} else {
/* manage mapping error */
}
} else {
/* manage prepare error */
}
The code sample above maps the allocated segment from its beginning (offset = 0) and for its entire size (size = RECEIVER_SEG_SIZE) to a local pointer. It is possible to map only parts of the segment, varying these two parameters, with the constraint that the sum of size and offset does not go beyond the end of the segment. If that happens, the function returns the error SCI_ERR_OUT_OF_RANGE. Size and offset must also satisfy some implementation-dependent alignment constraints; otherwise, error is set to either SCI_ERR_SIZE_ALIGNMENT or SCI_ERR_OFFSET_ALIGNMENT. If you need these implementation-dependent values, you can use SCIQuery() with command SCI_Q_ADAPTER and sub-commands SCI_Q_ADAPTER_DMA_SIZE_ALIGNMENT and SCI_Q_ADAPTER_DMA_OFFSET_ALIGNMENT.
suggested_address is a suggested virtual address where the segment should be mapped. Its value is taken into account only if the flag SCI_FLAG_FIXED_MAP_ADDR is set.
SCIMapLocalSegment() also accepts the flag SCI_FLAG_READONLY_MAP, which causes the segment to be mapped in read-only mode. If the function call succeeds, the returned value map_address is a pointer to the beginning of the mapped segment and map is a handle to a correctly initialized mapped segment descriptor.
Mapping a remote memory segment into addressable space¶
Mapping a remote memory segment into the addressable space of a program is very similar to the procedure followed for a local segment. The following code shows the usage of SCIMapRemoteSegment():
sci_desc_t v_dev;
sci_remote_segment_t remote_segment;
sci_map_t remote_map;
sci_error_t error;
volatile void* map_address;
SCIInitialize(...);
SCIOpen(&v_dev, ...);
SCIConnectSegment(v_dev,
&remote_segment,
RECEIVER_NODE_ID,
RECEIVER_SEG_ID,
ADAPTER_NO,
NO_CALLBACK,
NO_ARG,
SCI_INFINITE_TIMEOUT,
NO_FLAGS,
&error);
if (error == SCI_ERR_OK) {
/* the remote segment has been successfully connected */
size_t offset = 0;
size_t size = SCIGetRemoteSegmentSize(remote_segment);
void* suggested_address = 0;
map_address = SCIMapRemoteSegment(remote_segment,
&remote_map,
offset,
size,
suggested_address,
NO_FLAGS,
&error);
if (error == SCI_ERR_OK) {
/* the segment has been successfully mapped at virtual
* address map_address */
} else {
/* manage mapping error */
}
} else {
/* manage connection error */
}
The code sample above maps the connected segment from its beginning (offset = 0) and for its entire size (which has been determined using the function SCIGetRemoteSegmentSize() and should be equal to RECEIVER_SEG_SIZE). It is possible to only map parts of the segment, varying these two parameters, with the constraint that the sum of size and offset does not go beyond the end of the segment. If that happens, the function returns the error SCI_ERR_OUT_OF_RANGE. Size and offset must also satisfy some implementation-dependent alignment constraints; otherwise error is set to either SCI_ERR_SIZE_ALIGNMENT or SCI_ERR_OFFSET_ALIGNMENT.
suggested_address is a suggested virtual address where the segment should be mapped. Its value is taken into account only if the flag SCI_FLAG_FIXED_MAP_ADDR is set. SCIMapRemoteSegment() also accepts the flag SCI_FLAG_READONLY_MAP, which causes the segment to be mapped in read-only mode.
remote_map is a handle to the mapped segment descriptor and is initialized by the call, if successful. Note that the type of remote_map is sci_map_t, which is the same type of the handle to the local mapped segment descriptor.
If the function call succeeds, the returned value map_address is a pointer to the beginning of the mapped segment. Note that the pointer is declared as volatile to prevent the compiler from doing wrong optimizations of the code.
Unmapping a mapped memory segment¶
A mapped memory segment should be unmapped once it is no longer needed. Since the type of a local and a remote mapped segment is the same, there is a unique function, called SCIUnmapSegment(), that performs this operation.
For a local segment then, the sequence of operations from creation to removal, for what concerns local access, is the following:
sci_local_segment_t local_segment;
sci_map_t local_map;
sci_error_t error;
SCICreateSegment(..., &local_segment, ...);
SCIPrepareSegment(local_segment, ADAPTER_NO, ...);
SCIMapLocalSegment(local_segment, &local_map, ...);
/* use the mapped segment */
SCIUnmapSegment(local_map, NO_FLAGS, &error)
if (error == SCI_ERR_OK) {
/* the segment is not mapped any more */
} else {
/* manage error */
}
SCIRemoveSegment(local_segment, ...);
For a remote segment:
sci_remote_segment_t remote_segment;
sci_map_t remote_map;
sci_error_t errror;
void* addr;
SCIConnectSegment(..., &remote_segment, ...);
SCIMapRemoteSegment(remote_segment, &remote_map, ...);
/* use the mapped segment */
SCIUnmapSegment(remote_map, NO_FLAGS, &error);
if (error == SCI_ERR_OK) {
/* the remote segment is not mapped any more */
} else {
/* manage error */
}
SCIDisconnectSegment(remote_segment, ...);
For a remote segment to be connectable, it must have been properly exported on the other node. The syntax of SCIUnmapSegment() is very simple: local_map ( remote_map) is a handle to a mapped local (remote) segment descriptor; there are no special flags and, if the function fails, error typically has the value
SCI_ERR_BUSY, meaning that you have not correctly considered all the dependencies to the mapped segment.
Accessing data within mapped memory segments¶
Once a local or remote memory segment is mapped into the addressable space, a program sees that memory as any other piece of memory it has allocated using malloc() or similar functions and it can access it via the normal use of pointers: it can read from it, write to it, do a memcpy() and so on.
For a local segment:
sci_local_segment_t local_segment;
sci_map_t local_map;
int* l_addr; /* address to local segment */
SCICreateSegment(..., &local_segment, ...);
SCIPrepareSegment(local_segment, ADAPTER_NO, ...);
l_addr = (int*)SCIMapLocalSegment(local_segment, &local_map, ...);
*l_addr = 1; /* local write operation */
*l_addr; /* local read operation */
For a remote segment:
sci_remote_segment_t remote_segment;
sci_map_t remote_map;
volatile int* r_addr; /* address to remote segment */
SCIConnectSegment(..., &remote_segment, ...);
r_addr = (volatile int*)SCIMapRemoteSegment(remote_segment, &remote_map, ...);
*r_addr = 1; /* remote write operation */
*r_addr; /* remote read operation */
Remote operations enables access to memory that is physically resident on another node. Having mapped access to remote memory allows very low latency, very low overhead data transfers based on simple CPU load or store operations.
Remote memory access example¶
It’s time to summarize all the topics so far in a sort of “big picture”, in order to have a proper understanding of the available features offered by the SISCI API for the use of remote memory. There are indeed other important aspects, for example, how to check for data transfer errors, but we are already able to implement a simple “Hello, World!” application. This is actually composed of two programs: a sender sends a command to a receiver which then prints the string “Hello, World!”.
Example 4-1 shows the sender. It opens a virtual device, connects to a known remote segment, maps it into its own address space, and writes the print command at the beginning of that piece of memory, which is automatically converted by the hardware into a remote write operation, and finally cleans everything up and exits.
Example 4-1. A sender program based on remote memory access
/* sender program */
#include "sisci_api.h"
#define RECEIVER_NODE_ID 4
#define RECEIVER_SEG_ID 4
#define NO_ARG 0
#define NO_FLAGS 0
#define PRINT_COMMAND 1
int main(int argc, char* argv[])
{
sci_desc_t v_dev;
sci_error_t error;
sci_remote_segment_t remote_segment;
size_t remote_segment_size;
sci_map_t remote_map;
volatile int* remote_address;
/* initialize the SCI environment */
SCIInitialize(NO_FLAGS, &error);
if (error != SCI_ERR_OK) return 1;
/* create a virtual device */
SCIOpen(&v_dev, NO_FLAGS, &error);
if (error != SCI_ERR_OK) return 1;
/* connect to the remote segment */
SCIConnectSegment(v_dev, &remote_segment, RECEIVER_NODE_ID, RECEIVER_SEG_ID,
ADAPTER_NO, NO_CALLBACK, NO_ARG, SCI_INFINITE_TIMEOUT,
NO_FLAGS, &error);
if (error != SCI_ERR_OK) return 1;
remote_segment_size = SCIGetRemoteSegmentSize(remote_segment);
/* map the remote segment */
remote_address = (volatile int*)SCIMapRemoteSegment(remote_segment, &remote_map,
0 /* offset */,
remote_segment_size,
0 /* address hint */,
NO_FLAGS, &error);
if (error != SCI_ERR_OK) return 1;
/* send the print command */
*remote_address = PRINT_COMMAND;
/* cleanup */
SCIUnmapSegment(remote_map, NO_FLAGS, &error);
if (error != SCI_ERR_OK) return 1;
SCIDisconnectSegment(remote_segment, NO_FLAGS, &error);
if (error != SCI_ERR_OK) return 1;
SCIClose(v_dev, NO_FLAGS, &error);
if (error != SCI_ERR_OK) return 1;
SCITerminate();
return 0;
}
Example 4-2 shows the receiver. It opens a virtual device, allocates a local memory segment, maps it into its address space in such a way that it can access it, initializes the first word of that memory to something different than the print command, exports the segment into the network address space, waits for the print command (which simply means checking the first word of the memory segment in a polling loop), prints a message, and exits after having cleaned up.
Note that the initialization of the first word of the memory segment is done before the segment itself is made available to other nodes to avoid a race condition, that is, to ensure the value updated from remote is not overwritten by a late initialization.
Example 4-2. A receiver program based on remote memory access[]{#_Ref366677226 .anchor}
/* receiver program */
#include "sisci_error.h"
#include "sisci_api.h"
#include <stdio.h>
#define RECEIVER_SEG_ID 4
#define RECEIVER_SEG_SIZE 4096
#define ADAPTER_NO 0
#define NO_CALLBACK 0
#define NO_ARG 0
#define NO_FLAGS 0
#define PRINT_COMMAND 1
int main(int argc, char* argv[])
{
sci_desc_t v_dev;
sci_error_t error;
sci_local_segment_t local_segment;
sci_map_t local_map;
int* local_address;
/* initialize the SCI environment */
SCIInitialize(NO_FLAGS, &error);
if (error != SCI_ERR_OK) return 1;
/* create a virtual device */
SCIOpen(&v_dev, NO_FLAGS, &error);
if (error != SCI_ERR_OK) return 1;
/* allocate a local segment */
SCICreateSegment(v_dev, &local_segment,
RECEIVER_SEG_ID, RECEIVER_SEG_SIZE,
NO_CALLBACK, NO_ARG, NO_FLAGS, &error);
if (error != SCI_ERR_OK) return 1;
SciPrepareSegment(local_segment, ADAPTER_NO, NO_FLAGS, &error);
if (error != SCI_ERR_OK) return 1;
/* map the local segment */
local_address = (int*)SCIMapLocalSegment(local_segment, &local_map,
0 /* offset */,
RECEIVER_SEG_SIZE,
0 /* address hint */,
NO_FLAGS,
&error);
if (error != SCI_ERR_OK) return 1;
/* initialise the first word of the to-be- accessed segment */
*local_address = ~PRINT_COMMAND;
/* export local segment */
SCISetSegmentAvailable(local_segment, ADAPTER_NO, NO_FLAGS, &error);
if (error != SCI_ERR_OK) return 1;
/* wait for the sender process to send the print command */
while (*l_addr != PRINT_COMMAND) ;
printf("Hello, World!\\n");
/* cleanup */
SCISetSegmentUnavailable(local_segment, ADAPTER_NO, NO_FLAGS, &error);
if (error != SCI_ERR_OK) return 1;
SCIUnmapSegment(local_map, NO_FLAGS, &error)
if (error != SCI_ERR_OK) return 1;
SCIRemoveSegment(local_segment, NO_FLAGS, error);
if (error != SCI_ERR_OK) return 1;
SCIClose(v_dev, NO_FLAGS, &error);
if (error != SCI_ERR_OK) return 1;
SCITerminate();
return 0;
}