The send call described in Section Blocking Send
is blocking:
it does not return until the message data
and envelope have been safely stored away so that the sender is
free to modify
the send buffer.
The message might be copied directly into the matching receive buffer,
or it might be copied into a temporary system buffer.
Message buffering
decouples the send and receive operations. A blocking send can
complete as soon
as the message was buffered, even if no matching receive has been executed
by the receiver. On the other hand, message buffering can be expensive,
as it entails additional memory-to-memory copying, and it requires the
allocation of memory for buffering. MPI offers the choice of
several communication modes that allow one to control the choice of the
communication protocol.
The send call described in Section Blocking Send
uses
the standard communication mode. In this mode,
it is up to MPI to decide whether outgoing
messages will be buffered. MPI may
buffer outgoing messages. In such a case, the send call may complete
before a matching receive is invoked. On the other hand, buffer space may be
unavailable, or MPI may choose not to buffer
outgoing messages, for performance reasons. In this case,
the send call will not complete until a matching receive has been posted, and
the data has been moved to the receiver.
Thus, a send in standard mode can be started whether or not a
matching receive has been posted. It may complete before a matching receive
is posted. The
standard mode send is non-local: successful completion of the send
operation may depend on the occurrence of a matching receive.
The reluctance of MPI to mandate whether standard sends are buffering
or not stems from the desire to achieve portable programs. Since any
system will run out of buffer resources as message sizes are increased,
and some implementations may want to provide little buffering, MPI
takes the position that correct (and therefore, portable) programs
do not rely on system buffering in standard mode. Buffering
may improve the performance
of a correct program, but it doesn't affect the result of the program.
If the user wishes to guarantee a certain amount of buffering, the
user-provided buffer system of Section Buffer Allocation and Usage
should be used,
along with the buffered-mode send.
( End of rationale.)
There are three additional communication modes.
A buffered mode send operation can be started whether or not a
matching receive has been posted.
It may complete before a matching receive is posted. However, unlike
the standard send, this operation is local, and its
completion does not depend on the occurrence of a matching receive. Thus, if a
send is executed and no matching receive is posted, then MPI must buffer the
outgoing message, so as to allow the send call to complete. An error will
occur if there is insufficient buffer space. The amount of available buffer
space is controlled by the user --- see Section Buffer Allocation and Usage
.
Buffer allocation by the user may be required for the buffered mode to be
effective.
A send that uses the synchronous mode can be started whether or
not a matching receive was posted. However, the send will complete
successfully only if a matching receive is posted, and the
receive operation has started to receive the message sent by the
synchronous send.
Thus, the completion of a synchronous send not only indicates that the
send buffer can be reused, but
it
also indicates that the receiver has
reached a certain point in its execution, namely that it has started
executing the matching receive. If both sends and receives are
blocking operations then the use of the synchronous mode provides
synchronous communication semantics: a communication does not complete
at either end before both processes rendezvous at the
communication. A send executed in this mode is non-local.
A send that uses the ready communication mode
may be started only if the matching receive is already posted.
Otherwise, the operation is erroneous and its outcome is undefined.
On some systems, this allows the removal of a hand-shake
operation that is otherwise required and results in improved
performance.
The completion of the send operation does not depend on the
status of a matching receive, and merely indicates that the send
buffer can be reused. A send operation that uses the ready mode has
the same semantics as a standard send operation, or a synchronous send
operation; it is merely that the sender provides additional
information to the system (namely that a matching receive is already
posted), that can save some overhead. In a correct program, therefore, a
ready send could be replaced by a standard send with no effect on the
behavior of the program other than performance.
Three additional send functions are provided for the three additional
communication
modes. The communication mode is indicated by a one letter prefix:
B for buffered,
S for synchronous, and
R for ready.
int MPI_Bsend(void* buf, int count, MPI_Datatype datatype, int dest, int tag, MPI_Comm comm)
MPI_BSEND(BUF, COUNT, DATATYPE, DEST, TAG, COMM, IERROR)
int MPI_Ssend(void* buf, int count, MPI_Datatype datatype, int dest, int tag, MPI_Comm comm)
MPI_SSEND(BUF, COUNT, DATATYPE, DEST, TAG, COMM, IERROR)
int MPI_Rsend(void* buf, int count, MPI_Datatype datatype, int dest, int tag, MPI_Comm comm)
MPI_RSEND(BUF, COUNT, DATATYPE, DEST, TAG, COMM, IERROR)
There is only one receive operation,
but it matches
any of the send modes.
The receive operation described in the last section is blocking:
it returns only after the receive buffer contains the newly received
message. A receive can complete before the matching send
has completed (of course, it can complete only after the matching send
has started).
In a multi-threaded implementation of MPI, the system may de-schedule a
thread that is blocked on a send or receive
operation, and schedule another thread for execution in the same
address space.
In such a case it is the user's responsibility not to
modify a
communication
buffer until the communication completes.
Otherwise, the outcome of the computation is undefined.
2.2
Since a synchronous send cannot complete before a matching receive is posted,
one will not normally buffer messages sent by such an operation.
It is recommended to choose buffering over blocking the sender, whenever
possible, for standard sends. The programmer can signal his or her preference
for blocking the sender until a matching receive occurs by using the
synchronous send mode.
A possible communication protocol for the various communication modes
is outlined below.
ready send: The message is sent as soon as possible.
synchronous send:
The sender sends a request-to-send message.
The receiver stores this request.
When a matching receive is posted, the receiver sends back a permission-to-send
message, and the sender now sends the message.
standard send:
First protocol may be used for short messages, and second protocol for
long messages.
buffered send:
The sender copies the message into a buffer and then sends it with a
nonblocking send (using the same protocol as for standard send).
Additional control messages might be needed for flow control and error
recovery. Of course, there are many other possible protocols.
Ready send can be implemented as a standard send. In this case
there will be no performance advantage (or disadvantage) for the use of
ready send.
A standard send can be implemented as a synchronous send.
In such a case, no data buffering is needed. However,
users may
expect some buffering.
In a multi-threaded environment, the execution of a blocking communication
should block only the executing thread, allowing the thread scheduler to
de-schedule this thread and schedule another thread for execution.
( End of advice to implementors.)
Rationale.
MPI_BSEND (buf, count, datatype, dest, tag, comm) IN buf initial address of send buffer (choice) IN count number of elements in send buffer (non-negative
integer) IN datatype datatype of each send buffer element (handle) IN dest rank of destination (integer) IN tag message tag (integer) IN comm communicator (handle)
<type> BUF(*)
INTEGER COUNT, DATATYPE, DEST, TAG, COMM, IERROR
{ void MPI::Comm::Bsend(const void* buf, int count, const MPI::Datatype& datatype, int dest, int tag) const (binding deprecated, see Section Deprecated since MPI-2.2
) }
Send in buffered mode.
MPI_SSEND (buf, count, datatype, dest, tag, comm) IN buf initial address of send buffer (choice) IN count number of elements in send buffer (non-negative
integer) IN datatype datatype of each send buffer element (handle) IN dest rank of destination (integer) IN tag message tag (integer) IN comm communicator (handle)
<type> BUF(*)
INTEGER COUNT, DATATYPE, DEST, TAG, COMM, IERROR
{ void MPI::Comm::Ssend(const void* buf, int count, const MPI::Datatype& datatype, int dest, int tag) const (binding deprecated, see Section Deprecated since MPI-2.2
) }
Send in synchronous mode.
MPI_RSEND (buf, count, datatype, dest, tag, comm) IN buf initial address of send buffer (choice) IN count number of elements in send buffer (non-negative
integer) IN datatype datatype of each send buffer element (handle) IN dest rank of destination (integer) IN tag message tag (integer) IN comm communicator (handle)
<type> BUF(*)
INTEGER COUNT, DATATYPE, DEST, TAG, COMM, IERROR
{ void MPI::Comm::Rsend(const void* buf, int count, const MPI::Datatype& datatype, int dest, int tag) const (binding deprecated, see Section Deprecated since MPI-2.2
) }
Send in ready mode.
Advice
to implementors.
Up: Contents
Next: Semantics of Point-to-Point Communication
Previous: Data Conversion
Return to MPI-2.2 Standard Index
Return to MPI Forum Home Page
(Unofficial) MPI-2.2 of September 4, 2009
HTML Generated on September 10, 2009