The send call described in Section Blocking Send is blocking: it does not return until the message data and envelope have been safely stored away so that the sender is free to modify the send buffer. The message might be copied directly into the matching receive buffer, or it might be copied into a temporary system buffer.
Message buffering decouples the send and receive operations. A blocking send can complete as soon as the message was buffered, even if no matching receive has been executed by the receiver. On the other hand, message buffering can be expensive, as it entails additional memory-to-memory copying, and it requires the allocation of memory for buffering. MPI offers the choice of several communication modes that allow one to control the choice of the communication protocol.
The send call described in Section Blocking Send uses the standard communication mode. In this mode, it is up to MPI to decide whether outgoing messages will be buffered. MPI may buffer outgoing messages. In such a case, the send call may complete before a matching receive is invoked. On the other hand, buffer space may be unavailable, or MPI may choose not to buffer outgoing messages, for performance reasons. In this case, the send call will not complete until a matching receive has been started, and the data has been moved to the receiver.
Thus, a standard mode send can be started whether or not a matching receive has been started. It may complete before a matching receive is started. The standard mode send is nonlocal: successful completion of the send operation may depend on the occurrence of a matching receive.
Rationale.
The reluctance of MPI to mandate whether standard sends are buffering
or not stems from the desire to achieve portable programs. Since any
system will run out of buffer resources as message sizes are increased,
and some implementations may want to provide little buffering, MPI
takes the position that correct (and therefore, portable) programs
do not rely on system buffering in standard mode. Buffering
may improve the performance
of a correct program, but it doesn't affect the result of the program.
If the user wishes to guarantee a certain amount of buffering, the
user-provided buffer system of Section Buffer Allocation and Usage should be used,
along with the buffered-mode send.
( End of rationale.)
There are three additional communication modes.
A buffered mode send operation can be started whether or not a matching receive has been started. It may complete before a matching receive is started. However, unlike the standard send, this operation is local, and its completion does not depend on the occurrence of a matching receive. Thus, if a send is executed and no matching receive is started, then MPI must buffer the outgoing message, so as to allow the send call to complete. An error will occur if there is insufficient buffer space. The amount of available buffer space is controlled by the user---see Section Buffer Allocation and Usage. Buffer allocation by the user may be required for the buffered mode to be effective.
A send that uses the synchronous mode can be started whether or not a matching receive was started. However, the send will complete successfully only if a matching receive is started, and the receive operation has started to receive the message sent by the synchronous send. Thus, the completion of a synchronous send not only indicates that the send buffer can be reused, but it also indicates that the receiver has reached a certain point in its execution, namely that it has started executing the matching receive. If both sends and receives are blocking operations then the use of the synchronous mode provides synchronous communication semantics: a communication does not complete at either end before both MPI processes rendezvous at the communication. A send executed in this mode is nonlocal.
A send that uses the ready communication mode may be started only if the matching receive is already started. Otherwise, the operation is erroneous and its outcome is undefined. On some systems, this allows the removal of a hand-shake protocol that is otherwise required and results in improved performance. The completion of the send operation does not depend on the status of a matching receive, and merely indicates that the send buffer can be reused. A send operation that uses the ready mode has the same semantics as a standard send operation, or a synchronous send operation; it is merely that the sender provides additional information to the system (namely that a matching receive is already started), that can save some overhead. In a correct program, therefore, a ready send could be replaced by a standard send with no effect on the behavior of the program other than performance.
Three additional send functions are provided for the three additional communication modes. The communication mode is indicated by a one letter prefix: B for buffered, S for synchronous, and R for ready.
| MPI_BSEND(buf, count, datatype, dest, tag, comm) | |
| IN buf | initial address of send buffer (choice) |
| IN count | number of elements in send buffer (non-negative integer) |
| IN datatype | datatype of each send buffer element (handle) |
| IN dest | rank of destination (integer) |
| IN tag | message tag (integer) |
| IN comm | communicator (handle) |
Send in buffered mode.
According to the definitions in Section MPI Procedures,
MPI_BSEND is a completing procedure and the user can re-use
all resources given as arguments, including the message data buffer.
It is also a local procedure because it returns immediately without
depending on the execution of any MPI procedure in any other MPI process.
Advice to users.
This is one of the exceptions in which a completing and therefore blocking operation-related procedure is local.
( End of advice to users.)
| MPI_SSEND(buf, count, datatype, dest, tag, comm) | |
| IN buf | initial address of send buffer (choice) |
| IN count | number of elements in send buffer (non-negative integer) |
| IN datatype | datatype of each send buffer element (handle) |
| IN dest | rank of destination (integer) |
| IN tag | message tag (integer) |
| IN comm | communicator (handle) |
Send in synchronous mode.
| MPI_RSEND(buf, count, datatype, dest, tag, comm) | |
| IN buf | initial address of send buffer (choice) |
| IN count | number of elements in send buffer (non-negative integer) |
| IN datatype | datatype of each send buffer element (handle) |
| IN dest | rank of destination (integer) |
| IN tag | message tag (integer) |
| IN comm | communicator (handle) |
Send in ready mode.
There is only one receive operation, but it matches any of the send modes. The receive procedure described in the last section is blocking: it returns only after the receive buffer contains the newly received message. A receive can complete before the matching send has completed (of course, it can complete only after the matching send has started).
In a multithreaded implementation of MPI, the system may de-schedule a thread that is blocked on a send or receive operation, and schedule another thread for execution in the same address space. In such a case it is the user's responsibility not to modify a communication buffer until the communication completes. Otherwise, the outcome of the computation is undefined.
Advice
to implementors.
Since a synchronous send cannot complete before a matching receive is started, one will not normally buffer messages sent by such an operation.
It is recommended to choose buffering over blocking the sender, whenever possible, for standard sends. The programmer can signal a preference for blocking the sender until a matching receive occurs by using the synchronous send mode.
A possible communication protocol for the various communication modes is outlined below.
Ready send can be implemented as a standard send. In this case there will be no performance advantage (or disadvantage) for the use of ready send.
A standard send can be implemented as a synchronous send. In such a case, no data buffering is needed. However, users may expect some buffering.
In a multithreaded environment, the execution of a blocking communication
should block only the executing thread, allowing the thread scheduler to
de-schedule this thread and schedule another thread for execution.
( End of advice to implementors.)