Advanced features¶
This chapter addresses some issues that are slightly more advanced than the basic features presented up till now. Nonetheless, they are fundamental for writing a complete SISCI application.
In this chapter, you will learn:
How to exploit caching techniques to improve performance.
How to check for data transfer errors and cope with them.
How to exploit events generated by the underlying adapter card and interconnect.
How to use the available callback mechanism in a number of situations.
Error checking¶
Remote data access is more error-prone than local access. Although the network protocol is robust, an error can always happen, due for example to a cable being unplugged or a switch being power cycled. Error situations detected by the network hardware are made available to the upper software layers, which can then react appropriately; for example, an application may decide to ignore the error while another may retry the affected data transfer or switch over to an alternative network. The SISCI API provides the means to cope with both buffering and with data transfer errors. In both cases, the solution is based on the concept of a “sequence”.
Sequences¶
A sequence is a resource associated to a remote mapped segment, which allows error checking in data transfers to that segment. The name “sequence” comes from the fact that you are supposed to use it when you execute a sequence of read or write operations to a remote segment. Like all the resources, a sequence has its own descriptor type (sci_sequence) and its own handle type (sci_sequence_t).
A sequence resource is allocated using the API function SCICreateMapSequence().
sci_map_t remote_map;
sci_sequence_t sequence;
sci_error_t error;
SCIMapRemoteSegment(..., &remote_map, ...);
SCICreateMapSequence(remote_map, &sequence, NO_FLAGS, &error);
if (error == SCI_ERR_OK) {
/* the sequence is available */
} else {
/* manage error */
}
remote_map represents a remote segment which has been memory-mapped locally. Sequence is the handle to the sequence descriptor that is allocated and initialized by the function call. Once a sequence is not used anymore, it is destroyed by invoking SCIRemoveSequence():
sci_sequence_t sequence;
sci_error_t error;
SCICreateMapSequence(..., &sequence, ...);
/* use the sequence */
SCIRemoveSequence(sequence, NO_FLAGS, &error);
if (error == SCI_ERR_OK) {
/* the sequence resource is released */
} else {
/* manage error */
}
Flushing buffers¶
As mentioned above, one may want to flush write buffers to make sure the data transfer has started to the remote memory. SCIFlush() is provided to clear any write buffers, the data may still be in transit to the remote memory when the function returns:
sci_sequence_t sequence;
sci_map_t remote_map;
volatile void* map_address;
map_address = SCIMapRemoteSegment(..., &remote_map, ...);
SCICreateMapSequence(remote_map, &sequence, ...);
*map_address = 1; /* writes the value 1 to remote memory */
SCIFlush(sequence);
SCIStoreBarrier() is used to empty the write buffers and force any transfer packets to be transmitted and wait for its completion:
sci_sequence_t sequence;
sci_map_t remote_map;
volatile void* map_address;
map_address = SCIMapRemoteSegment(..., &remote_map, ...);
SCICreateMapSequence(remote_map, &sequence, ...);
*map_address = 1; /* remote write operation */
SCIStoreBarrier(sequence, NO_FLAGS);
SCIStoreBarrier() returns when transactions related to the associated segment have completed. In other words, when SCIStoreBarrier() returns, you are sure that all the possibly buffered data for that segment has arrived at the receiver’s memory unless there has been a transmission error. You should use SCICheckSequence() if you want to include a check for transmission errors.
Note that SCIFlush() is a local-only operation, affecting only write buffers on the local adapter card, while SCIStoreBarrier() usually causes some data transmission and therefore some interaction with remote nodes. SCIStoreBarrier() and SCICheckSequence() are relatively costly operations, so you may not want to design your application to use these extensively.
Checking for data transfer errors¶
Sequence is also used to check for errors during data transfers. A sequence needs to be cleared before starting to move data. This means that no pending errors should exist for the concerned segment.
sci_sequence_t sequence;
sci_error_t error;
sci_sequence_status_t status;
sci_map_t remote_map;
SCICreateMapSequence(remote_map, &sequence, ...);
status = SCIStartSequence(sequence, NO_FLAGS, &error);
if (error == SCI_ERR_OK) {
/* check the status */
} else {
/* manage error */
}
The return value of SCIStartSequence(), if successful, determines whether the data transfer can start (status equal to SCI_SEQ_OK) or if there are some pending errors (status equal to SCI_SEQ_PENDING). In the latter case, the function must be called again. The type sci_sequence_status_t contains four different values, but the other two are meaningless for SCIStartSequence().
The code below demonstrates the use of SCIStartSequence() before the data transfer:
sci_error_t error;
sci_sequence_t sequence;
sci_sequence_status_t status;
do {
status = SCIStartSequence(sequence, NO_FLAGS, &error);
if (error != SCI_ERR_OK) {
/* manage error */
}
} while (status != SCI_SEQ_OK);
/* can transfer data */
SCICheckSequence() checks if a data transfer was affected by errors. What this function actually does is to check that no errors have occurred since the last successful check, which could have been performed by either
SCICheckSequence() or SCIStartSequence().
sci_sequence_t sequence;
sci_error_t error;
sci_sequence_status_t status;
do {
status = SCIStartSequence(sequence, NO_FLAGS, &error);
if (error != SCI_ERR_OK) {
/* manage error */
}
} while (status != SCI_SEQ_OK)
/* transfer data */
status = SCICheckSequence(sequence, NO_FLAGS, &error);
if (error == SCI_ERR_OK) {
/* check the status */
} else {
/* manage errors */
}
status can assume all the four possible values of the sci_sequence_status_t type:
SCI_SEQ_OK: the transfer was error free.
SCI_SEQ_RETRIABLE: the transfer failed due to a non-fatal error (e.g. system busy because of heavy traffic) but can be immediately retried.
SCI_SEQ_NOT_RETRIABLE: the transfer failed due to a fatal error (e.g. cable unplugged) and can be retried only after a successful call to SCIStartSequence().
SCI_SEQ_PENDING: the transfer failed but the driver hasn’t been able yet to determine the severity of the error (if fatal or non-fatal); SCIStartSequence() must be called until it succeeds.
By default, SCICheckSequence() also flushes the write buffers and waits for all the interconnect transactions to complete; in other words, it internally performs an action similar to what SCIStoreBarrier() does. If you don’t want the flush to be performed, pass SCI_FLAG_NO_FLUSH as a flag. Alternatively, you can perform the flush but not wait for the completion of all the outstanding network transactions; in this case pass SCI_FLAG_NO_STORE_BARRIER as a flag. They can be OR’ed together if you want to pass both the flags.
Therefore, if you want an error-free data transfer you should:
Start the sequence.
Transfer the data.
Check the sequence.
a) It is ok continue.
b) If it is not ok and the error is not fatal, retry the transfer and check the sequence.
c) If it is not ok and the error is fatal, restart the sequence, retry the transfer and check the sequence.
The corresponding code would be something like the following:
sci_error_t error;
sci_sequence_t sequence;
sci_sequence_status_t status;
do {
status = SCIStartSequence(sequence, ..., &error);
if (error != SCI_ERR_OK) {
/* manage error */
}
} while (status!= SCI_SEQ_OK);
/* transfer data */
status = SCICheckSequence(sequence, ..., &error);
if (error != SCI_ERR_OK) {
/* manage error */
}
while (status != SCI_SEQ_OK) {
switch (status) {
case SCI_SEQ_RETRIABLE:
break;
case SCI_SEQ_PENDING:
case SCI_SEQ_NOT_RETRIABLE:
/* need a successful SCIStartSequence() before retrying */
do {
status = SCIStartSequence(sequence, ..., &error);
if (error != SCI_ERR_OK) {
/* manage error */
}
} while (status != SCI_SEQ_OK);
break;
default:
/* shouldn't happen; manage error */
}
/* transfer data */
status = SCICheckSequence(sequence, ..., &error);
if (error != SCI_ERR_OK) {
/* manage error */
}
}
If your application does not require reliable data transfers, but you still want to log when an error occurs, you could use something like the following:
sci_error_t error;
sci_sequence_t sequence;
sci_sequence_status_t status;
do {
status = SCIStartSequence(sequence, ..., &error);
if (error != SCI_ERR_OK) {
/* manage error */
}
} while (status != SCI_SEQ_OK);
/* transfer data */
status = SCICheckSequence(sequence, ..., &error);
if (error != SCI_ERR_OK) {
/* manage error */
}
while (status != SCI_SEQ_OK) {
switch (status) {
case SCI_SEQ_RETRIABLE:
break;
case SCI_SEQ_PENDING:
case SCI_SEQ_NOT_RETRIABLE:
/* need a successful SCIStartSequence() before retrying */
do {
status = SCIStartSequence(sequence, ..., &error);
if (error != SCI_ERR_OK) {
/* manage error */
}
} while (status != SCI_SEQ_OK);
break;
default:
/* shouldn't happen; manage error */
}
/* log the occurence of the error */
status = SCICheckSequence(sequence, ..., &error);
if (error != SCI_ERR_OK) {
/* manage error */
}
}
Notice how the code is almost identical to the previous example for reliable communication, the only difference being that you now do not retry the data transfer after a failure; instead, you simply log somewhere that an error occurred. You still need to restart the sequence in case of error, otherwise the error flags would not be in a clean state at the beginning of the next data transfer.
Events and callbacks¶
On certain conditions a component of the network - be it a piece of hardware, a driver, or an application - generates a so-called event. Examples of an event are a cable being plugged or unplugged, the disappearance of a remote segment because of a node failure, the completion of a DMA queue processing, a triggered interrupt, etc.
Some events are managed directly by the SISCI driver, whereas others can be forwarded to an application, which can then either ignore or catch them. There are two ways you can catch an event: either you wait for it, blocking the process, or you can register a callback function, which will be called when the event occurs.
Each of the following sections concerns a different context for events: local memory segments, remote memory segments, DMA queues, and interrupts.
Local segment events¶
Five events are related to local memory segments. Their names are collected in an enumeration type called
sci_segment_cb_reason_t. The name comes from the fact that a member of such enumeration is passed to a callback function as the reason for its invocation (see later in the section).
typedef enum {
SCI_CB_CONNECT,
SCI_CB_DISCONNECT,
SCI_CB_NOT_OPERATIONAL,
SCI_CB_OPERATIONAL,
SCI_CB_LOST
} sci_segment_cb_reason_t;
Their meaning is the following:
SCI_CB_CONNECT
A new connection has been established from a remote node.
SCI_CB_DISCONNECT
An existing connection has been released.
SCI_CB_NOT_OPERATIONAL
The route to a connected node is temporarily unavailable.
SCI_CB_OPERATIONAL
The route to a connected node is available (again).
SCI_CB_LOST
An unrecoverable event has occurred on a connected node (e.g., it may have rebooted).
All these events are passed to the application, which may wish to intercept them. All the resources depending on a local segment should be freed before the segment itself could be removed. Some of these dependencies are due to connected nodes, so for example, the exporting application can catch the above events, in particular the CONNECT, DISCONNECT and LOST events, in order to keep an up-to-date table of all the established connections, along with their state.
Local segment synchronous event handling¶
The simplest way to catch events is just to wait for them:
sci_local_segment_t segment;
unsigned int source_node_id;
unsigned int local_adapter_no;
sci_error_t error;
sci_segment_cb_reason_t reason;
SCICreateSegment(..., &segment, ..., NO_FLAGS, ...);
SCIPrepareSegment(..., segment, ...);
SCISetSegmentAvailable(..., segment, ...);
/* The segment is now available for remote connections */
reason = SCIWaitForLocalSegmentEvent(segment,
&source_node_id,
&local_adapter_no,
SCI_INFINITE_TIMEOUT,
NO_FLAGS,
&error);
if (error == SCI_ERR_OK) {
switch (reason) {
case SCI_CB_CONNECT:
/* update the connection table for the segment */
break;
case SCI_CB_DISCONNECT:
/* update the connection table for the segment */
break;
case SCI_CB_OPERATIONAL:
/* the segment is usable */
break;
case SCI_CB_NOT_OPERATIONAL:
/* the situation may recover */
break;
case SCI_CB_LOST:
/* update the connection table for the segment */
break;
default:
/* error */
break;
}
} else {
/* manage error */
}
The code sample above makes a local segment available to remote nodes and waits for an event concerning the segment. The call to SCIWaitForLocalSegmentEvent() provides the event that caused it to return; moreover, it sets the identifier of the node that generated the event and the local adapter that received that event.
SCIWaitForLocalSegmentEvent() can fail because the timeout has expired (with error set to
SCI_ERR_TIMEOUT) or because the handle is invalid, for example when the segment has been removed (error is set to SCI_ERR_CANCELLED).
Local segment asynchronous event handling¶
An event can be caught asynchronously, through a callback mechanism. In order to do this, a callback function must be registered when calling SCICreateSegment(). Compare the following excerpt of C code with the one used in Managing local segments.
sci_desc_t v_dev;
sci_local_segment_t segment;
sci_error_t error;
void* arg = 0;
SCIInitialize(...);
SCIOpen(&v_dev,...);
/* possible setting of arg */
SCICreateSegment(v_dev, &segment,
RECEIVER_SEG_ID, /* segment identifier */
RECEIVER_SEG_SIZE, /* size */
local_segment_cb, /* callback function */
arg, /* callback argument */
SCI_FLAG_USE_CALLBACK, /* enables callback */
&error);
if (error == SCI_ERR_OK) {
/* a segment is available for use */
} else {
/* manage error */
}
Where local_segment_cb is a function with the following prototype:
sci_callback_action_t local_segment_cb(void* arg,
sci_local_segment_t segment,
sci_segment_cb_reason_t reason,
unsigned int node_id,
unsigned int local_adapter_no,
sci_error_t error);
Which corresponds to the type sci_cb_local_segment_t.
In the callback prototype, arg is the same parameter as specified in SCICreateSegment(). It is defined as a void* so that anything can be passed to it: from a null pointer, to a primitive value, to a pointer to a larger data structure.
The other parameters guarantee that the function gets the same information as is available after a SCIWaitForLocalSegmentEvent() call.
Note how the value of the flags parameter in SCICreateSegment() is now SCI_FLAG_USE_CALLBACK.
According to the API specification, the use of this option prevents us from using SCIWaitForLocalSegmentEvent().
The return value of a callback is of type sci_callback_action_t which is defined as follows:
typedef enum {
SCI_CALLBACK_CANCEL = 1,
SCI_CALLBACK_CONTINUE
} sci_callback_action_t;
The return value of the callback function tells the driver whether the callback is still active
(SCI_CALLBACK_CONTINUE) or not (SCI_CALLBACK_CANCEL) after the function execution.
A possible implementation of the callback function could simply include the switch statement introduced above, where arg can be, for example, the connection table:
sci_callback_action_t local_segment_cb(void* arg,
sci_local_segment_t segment,
sci_segment_cb_reason_t reason,
unsigned int node_id,
unsigned int local_adapter_no)
{
/* convert arg to a pointer to a connection table */
switch (reason) {
case SCI_CB_CONNECT:
/* update the connection table for the segment */
break;
case SCI_CB_DISCONNECT:
/* update the connection table for the segment */
return SCI_CALLBACK_CANCEL;
case SCI_CB_OPERATIONAL:
/* the segment is usable */
break;
case SCI_CB_NOT_OPERATIONAL:
/* the situation may recover */
break;
case SCI_CB_LOST:
/* update the connection table for the segment */
return SCI_CALLBACK_CANCEL;
default:
/* error */
break;
}
return SCI_CALLBACK_CONTINUE;
}
Remote segment events¶
The same five events related to local memory segments also applies to remote segments, though their meaning is different. Let us recall their enumeration type sci_segment_cb_reason_t:
typedef enum {
SCI_CB_CONNECT,
SCI_CB_DISCONNECT,
SCI_CB_NOT_OPERATIONAL,
SCI_CB_OPERATIONAL,
SCI_CB_LOST
} sci_segment_cb_reason_t;
Their meaning for a remote segment is:
SCI_CB_CONNECT
An asynchronous connection has completed successfully (see Managing remote segments).
SCI_CB_DISCONNECT
An asynchronous connection has failed (see Managing remote segments).
SCI_CB_NOT_OPERATIONAL
The route to the exporting node is temporarily unavailable.
SCI_CB_OPERATIONAL
The route to the exporting node is available (again).
SCI_CB_LOST
An unrecoverable situation has occurred on the exporting node.
Asynchronous connection¶
In Connecting to a remote segment, we have seen how to connect to a remote segment in a synchronous way, that is, we wait until SCIConnectSegment() returns, either because the connection has completed or because a timeout has expired.
Alternatively, it is possible to only initiate the connection and wait for the completion while doing other things. This is possible calling SCIConnectSegment() with the flag SCI_FLAG_ASYNCHRONOUS_CONNECT:
sci_remote_segment_t remote_segment;
sci_error_t error;
SCIConnectSegment(..., &segment, ..., SCI_FLAG_ASYNCHRONOUS_CONNECT, &error);
if (error == SCI_ERR_OK) {
/* the handle is valid, but the remote segment
* is NOT necessarily connected */
} else {
/* manage error */
}
In this case the SCIConnectSegment() returns immediately with a valid handle. Be careful, as a valid handle does not mean a valid descriptor. If the connection request is satisfied, the descriptor will be validated later, once the driver has filled all the fields with proper values. If the connection is refused, the descriptor will never become valid.
In both cases, the driver generates an appropriate event (SCI_CB_CONNECT and SCI_CB_DISCONNECT respectively) to notify the result to the application, which then has to react accordingly. In particular, in case of failure it has to release the descriptor, even if invalid, with SCIDisconnectSegment() (see Disconnecting from a remote segment)
Remote segment synchronous event handling¶
As for a local segment resource, remote segments events can be caught either synchronously, with a wait function, or asynchronously, enabling the callback mechanism.
Let us start with the synchronous approach:
sci_remote_segment_t segment;
sci_error_t status;
sci_error_t error;
SCIConnectSegment(..., &segment, ...);
reason = SCIWaitForRemoteSegmentEvent(segment, &status,
SCI_INFINITE_TIMEOUT,
NO_FLAGS, &error);
if (error == SCI_ERR_OK) {
switch (reason) {
case SCI_CB_CONNECT:
/* the previous call to SCIConnectSegment() has succeded;
* the handle to the descriptor is now usable
* this case makes sense only if SCIConnectSegment() was
* called with the flag SCI_FLAG_ASYNCHRONOUS_CONNECT */
break;
case SCI_CB_DISCONNECT:
/* SCISetSegmentUnavailable() has been called
* on the exporting node with SCI_FLAG_NOTIFY;
* we should clean up and disconnect
* clean up */
SCIDisconnectSegment(segment, NO_FLAGS, &error);
break;
case SCI_CB_OPERATIONAL:
/* the connection is established (or re-established); is usable */
break;
case SCI_CB_NOT_OPERATIONAL:
/* wait for the connection to recover */
break;
case SCI_CB_LOST:
/* the connection is lost, the segment is not usable any more */
break;
default:
/* error */
break;
}
} else {
/* manage error */
}
SCIWaitForRemoteSegmentEvent() fails if the timeout expires (with error set to SCI_ERR_TIMEOUT) or if the segment has already been disconnected, so that the handle is not valid anymore (error is set to
SCI_ERR_CANCELLED). In case the event is SCI_CB_LOST, it is responsibility of the application to clean things up, so that resources, for example descriptors, be appropriately released. So the segment, if mapped, has to be unmapped with SCIUnmapSegment() and then it has to be disconnected with SCIDisconnectSegment().
Remote segment asynchronous event handling¶
For remote segment events there is also the alternative to catch them asynchronously by specifying an appropriate callback function in the call to SCIConnectSegment(). Compare the following code with the one shown in Connecting to a remote segment.
sci_desc_t v_dev;
sci_remote_segment_t remote_segment;
sci_error_t error;
void* arg = 0;
SCIInitialize(...);
SCIOpen(&v_dev, ...);
/* possible setting of arg */
SCIConnectSegment(v_dev,
&remote_segment,
RECEIVER_NODE_ID,
RECEIVER_SEG_ID,
ADAPTER_NO,
remote_segment_cb, /* callback function */
arg, /* callback argument */
SCI_INFINITE_TIMEOUT,
SCI_FLAG_USE_CALLBACK, /* enables callback */
&error);
if (error == SCI_ERR_OK) {
/* the remote segment is connected */
} else {
/* manage error */
}
Where remote_segment_cb() has the following prototype:
sci_callback_action_t remote_segment_cb(void* arg,
sci_remote_segment_t remote_segment,
sci_segment_cb_reason_t reason,
sci_error_t status);
Which corresponds to the type sci_cb_remote_segment_t. In the callback prototype, arg is the same parameter specified in SCIConnectSegment(). It is defined as a void* so that it can be casted easily to any other type. arg can for example represent some parameters associated with segment, such as the remote node and segment identifier.
The other parameters guarantee that the function gets the same information as is available after SCIWaitForRemoteSegmentEvent().
Note how the value of the parameter flags in SCIConnectSegment() is now SCI_FLAG_USE_CALLBACK.
According to the API specification, the use of this option prevents us from using SCIWaitForRemoteSegmentEvent().
The return value of a callback is of type sci_callback_action_t which is defined as follows:
typedef enum {
SCI_CALLBACK_CANCEL = 1,
SCI_CALLBACK_CONTINUE
} sci_callback_action_t;
The return value of the callback function tells the driver whether the callback is still active
(SCI_CALLBACK_CONTINUE) or not (SCI_CALLBACK_CANCEL) after the function execution.
A possible implementation of remote_segment_cb() could for example include the switch statement shown in the previous section:
sci_callback_action_t remote_segment_cb(void* arg,
sci_remote_sement_t segment,
sci_segment_cb_reason_t reason,
sci_error_t status)
{
switch (reason) {
case SCI_CB_CONNECT:
/* an asynchronous connection, i.e. SCIConnectSegment()
* called with the flag SCI_FLAG_ASYNCHRONOUS_CONNECT, has succeeded */
break;
case SCI_CB_DISCONNECT:
/* SCISetSegmentUnavailable() has been called on the exporting node
* with SCI_FLAG_NOTIFY - we should clean up and disconnect */
SCIDisconnectSegment(segment, NO_FLAGS, &error);
break;
case SCI_CB_OPERATIONAL:
/* the connection is established (or re-established);
* the segment is usable */
break;
case SCI_CB_NOT_OPERATIONAL:
/* wait for the connection to recover */
break;
case SCI_CB_LOST:
/* the connection is lost, the segment is not usable any more */
break;
default:
/* error */
break;
}
return SCI_CALLBACK_CONTINUE;
}
A state machine for a remote segment¶
The remote segment callback state machine can be found in State machine for remote segment callbacks.
State machine for remote segment callbacks¶
DMA and callbacks¶
In Waiting for DMA completion we have explored two ways for checking if the processing of a DMA queue has completed. One is based on a blocking wait function; the other is based on polling the state of the queue until it is in a final state. In this section, you will learn how to use a third way, which consists of a callback mechanism: you declare that a certain function is to be called when the DMA queue processing has completed. You declare the callback function when you post the queue for processing:
sci_error_t error;
sci_dma_queue_t dma_queue;
void* arg = 0;
SCICreateDMAQueue(..., &dma_queue, ...);
SCIStartDMATransfer(dma_queue,....,dma_queue_cb, arg,
SCI_FLAG_USE_CALLBACK,&error );
if (error == SCI_ERR_OK) {
/* queue posted successfully */
} else {
/* manage error */
}
Where dma_queue_cb() has the following prototype:
sci_callback_action_t
dma_queue_cb(void* arg, sci_dma_queue_t dma_queue, sci_error_t status);
Which corresponds to the type sci_cb_dma_t. Notice also, that you must explicitly set SCI_FLAG_USE_CALLBACK as a flag for the callback function to be considered. Remember that if you use a callback you are not allowed to use SCIWaitForDMAQueue().
Interrupts and callbacks¶
In Waiting for an interrupt we have seen that interrupts can be caught synchronously by calling SCIWaitForInterrupt(). Alternatively, you can specify a function to be called asynchronously when an interrupt arrives, on creation time.
sci_desc_t v_dev;
sci_local_interrupt_t local_interrupt;
unsigned int interrupt_no;
sci_error_t error;
void* arg = 0;
SCIInitialize(...);
SCIOpen(&v_dev, ...);
/* possible setting of arg */
SCICreateInterrupt(v_dev,
&local_interrupt,
ADAPTER_NO,
interrupt_no,
interrupt_cb,
arg,
SCI_FLAG_USE_CALLBACK,
&error);
if (error == SCI_ERR_OK) {
/* the interrupt is available to remote applications */
} else {
/* manage error */
}
Where interrupt_cb() has the following prototype:
sci_callback_action_t interrupt_cb(void* arg,
sci_local_interrupt_t interrupt,
sci_error_t error);
Which corresponds to the type sci_cb_interrupt_t.
Remember that:
You must pass the SCI_FLAG_USE_CALLBACK for the callback to be considered.
If you use a callback you are not allowed to use SCIWaitForInterrupt().
Reflective memory – Multicast¶
Certain types of memory mapped interconnect solutions provide a hardware based multicast environment. Multicast functionality enables a single write transaction to reach multiple targets. A CPU store, DMA write or some other device that can do a posted write to the mapped multicast address can typically initiate the transaction.
Dolphin has extended the SISCI API to support multicast/reflective memory, currently available with the Dolphin Express DX, IX and PX and MX interconnect family. The multicast functionality is normally only available when your systems are connected using a network switch (The multicast is normally implemented in the switch).
The initialization of a multicast segment is very similar to initialization of regular remote segments.
The segment ID is used to specify the reflective memory ID and must be between 0 and the maximum number of reflective memory segments supported by the hardware (e.g. max of 4 for Dolphin Express IX).
To allocate, prepare or map a segment, the SCI_FLAG_BROADCAST flag must be set. You should use DIS_BROADCAST_NODEID_GROUP_ALL as nodeId, for remote operations that needs the remote nodeId as parameter.
The important functionality and parameters are:
nodeId = DIS_BROADCAST_NODEID_GROUP_ALL;
SCICreateSegment( ..., SCI_FLAG_BROADCAST, ...);
SCIPrepareSegment(..., SCI_FLAG_BROADCAST, ...);
SCIConnectSegment(..., nodeId , SCI_FLAG_BROADCAST, ...);
More information on multicast¶
The SISCI software distribution from Dolphin contains several source code examples demonstrating the real use of the SISCI reflective / Multicast functionality.
More details on reflective memory can be found in a separate white paper that can be downloaded from the from the whitepaper section on the Dolphin website.
Peer to peer transfers¶
The SISCI API supports registering of arbitrary physical addresses as SISIC segments and identifying physical addresses of remote mapped devices.
More information on peer to peer transfers¶
More details on peer to peer communication can be found in a separate white paper that can be downloaded from the whitepaper section on the Dolphin web site.