61. Probe and Cancel

The MPI_PROBE and MPI_IPROBE operations allow incoming messages to be checked for, without actually receiving them. The user can then decide how to receive them, based on the information returned by the probe (basically, the information returned by status). In particular, the user may allocate memory for the receive buffer, according to the length of the probed message.

The MPI_CANCEL operation allows pending communications to be canceled. This is required for cleanup. Posting a send or a receive ties up user resources (send or receive buffers), and a cancel may be needed to free these resources gracefully.

MPI_IPROBE(source, tag, comm, flag, status)
IN source	source rank, or MPI_ANY_SOURCE (integer)
IN tag	tag value or MPI_ANY_TAG (integer)
IN comm	communicator (handle)
OUT flag	(logical)
OUT status	status object (Status)

int MPI_Iprobe(int source, int tag, MPI_Comm comm, int *flag, MPI_Status *status)

MPI_IPROBE(SOURCE, TAG, COMM, FLAG, STATUS, IERROR)
LOGICAL FLAG
INTEGER SOURCE, TAG, COMM, STATUS(MPI_STATUS_SIZE), IERROR
bool MPI::Comm::Iprobe(int source, int tag, MPI::Status& status) const
bool MPI::Comm::Iprobe(int source, int tag) const

MPI_IPROBE(source, tag, comm, flag, status) returns flag = true if there is a message that can be received and that matches the pattern specified by the arguments source, tag, and comm. The call matches the same message that would have been received by a call to MPI_RECV(..., source, tag, comm, status) executed at the same point in the program, and returns in status the same value that would have been returned by MPI_RECV(). Otherwise, the call returns flag = false, and leaves status undefined.

If MPI_IPROBE returns flag = true, then the content of the status object can be subsequently accessed as described in Section Return Status to find the source, tag and length of the probed message.

A subsequent receive executed with the same communicator, and the source and tag returned in status by MPI_IPROBE will receive the message that was matched by the probe, if no other intervening receive occurs after the probe, and the send is not successfully cancelled before the receive.

If the receiving process is multi-threaded, it is the user's responsibility to ensure that the last condition holds.

The source argument of MPI_PROBE can be MPI_ANY_SOURCE, and the tag argument can be MPI_ANY_TAG, so that one can probe for messages from an arbitrary source and/or with an arbitrary tag. However, a specific communication context must be provided with the comm argument.

It is not necessary to receive a message immediately after it has been probed for, and the same message may be probed for several times before it is received.

MPI_PROBE(source, tag, comm, status)
IN source	source rank, or MPI_ANY_SOURCE (integer)
IN tag	tag value, or MPI_ANY_TAG (integer)
IN comm	communicator (handle)
OUT status	status object (Status)

int MPI_Probe(int source, int tag, MPI_Comm comm, MPI_Status *status)

MPI_PROBE(SOURCE, TAG, COMM, STATUS, IERROR)
INTEGER SOURCE, TAG, COMM, STATUS(MPI_STATUS_SIZE), IERROR
void MPI::Comm::Probe(int source, int tag, MPI::Status& status) const
void MPI::Comm::Probe(int source, int tag) const

MPI_PROBE behaves like MPI_IPROBE except that it is a blocking call that returns only after a matching message has been found.

The MPI implementation of MPI_PROBE and MPI_IPROBE needs to guarantee progress: if a call to MPI_PROBE has been issued by a process, and a send that matches the probe has been initiated by some process, then the call to MPI_PROBE will return, unless the message is received by another concurrent receive operation (that is executed by another thread at the probing process). Similarly, if a process busy waits with MPI_IPROBE and a matching message has been issued, then the call to MPI_IPROBE will eventually return flag = true unless the message is received by another concurrent receive operation.

Example

Use blocking probe to wait for an incoming message.

       CALL MPI_COMM_RANK(comm, rank, ierr) 
       IF (rank.EQ.0) THEN 
            CALL MPI_SEND(i, 1, MPI_INTEGER, 2, 0, comm, ierr) 
       ELSE IF (rank.EQ.1) THEN 
            CALL MPI_SEND(x, 1, MPI_REAL, 2, 0, comm, ierr) 
       ELSE IF (rank.EQ.2) THEN 
           DO i=1, 2 
              CALL MPI_PROBE(MPI_ANY_SOURCE, 0, 
                              comm, status, ierr) 
              IF (status(MPI_SOURCE) .EQ. 0) THEN 
100                CALL MPI_RECV(i, 1, MPI_INTEGER, 0, 0, comm, status, ierr) 
              ELSE 
200                CALL MPI_RECV(x, 1, MPI_REAL, 1, 0, comm, status, ierr) 
              END IF 
           END DO 
       END IF 

Example A similar program to the previous example, but now it has a problem.

       CALL MPI_COMM_RANK(comm, rank, ierr) 
       IF (rank.EQ.0) THEN 
            CALL MPI_SEND(i, 1, MPI_INTEGER, 2, 0, comm, ierr) 
       ELSE IF (rank.EQ.1) THEN 
            CALL MPI_SEND(x, 1, MPI_REAL, 2, 0, comm, ierr) 
       ELSE IF (rank.EQ.2) THEN 
           DO i=1, 2 
              CALL MPI_PROBE(MPI_ANY_SOURCE, 0, 
                              comm, status, ierr) 
              IF (status(MPI_SOURCE) .EQ. 0) THEN 
100                CALL MPI_RECV(i, 1, MPI_INTEGER, MPI_ANY_SOURCE, 
                                 0, comm, status, ierr) 
              ELSE 
200                CALL MPI_RECV(x, 1, MPI_REAL, MPI_ANY_SOURCE, 
                                 0, comm, status, ierr) 
              END IF 
           END DO 
       END IF 

Advice to implementors.

A call to MPI_PROBE(source, tag, comm, status) will match the message that would have been received by a call to MPI_RECV(..., source, tag, comm, status) executed at the same point. Suppose that this message has source s, tag t and communicator c. If the tag argument in the probe call has value MPI_ANY_TAG then the message probed will be the earliest pending message from source s with communicator c and any tag; in any case, the message probed will be the earliest pending message from source s with tag t and communicator c (this is the message that would have been received, so as to preserve message order). This message continues as the earliest pending message from source s with tag t and communicator c, until it is received. A receive operation subsequent to the probe that uses the same communicator as the probe and uses the tag and source values returned by the probe, must receive this message, unless it has already been received by another receive operation. ( End of advice to implementors.)

MPI_CANCEL(request)
IN request	communication request (handle)

int MPI_Cancel(MPI_Request *request)

MPI_CANCEL(REQUEST, IERROR)
INTEGER REQUEST, IERROR
void MPI::Request::Cancel() const

A call to MPI_CANCEL marks for cancellation a pending, nonblocking communication operation (send or receive). The cancel call is local. It returns immediately, possibly before the communication is actually canceled. It is still necessary to complete a communication that has been marked for cancellation, using a call to MPI_REQUEST_FREE, MPI_WAIT or MPI_TEST (or any of the derived operations).

If a communication is marked for cancellation, then a MPI_WAIT call for that communication is guaranteed to return, irrespective of the activities of other processes (i.e., MPI_WAIT behaves as a local function); similarly if MPI_TEST is repeatedly called in a busy wait loop for a canceled communication, then MPI_TEST will eventually be successful.

MPI_CANCEL can be used to cancel a communication that uses a persistent request (see Section Persistent Communication Requests ), in the same way it is used for nonpersistent requests. A successful cancellation cancels the active communication, but not the request itself. After the call to MPI_CANCEL and the subsequent call to MPI_WAIT or MPI_TEST, the request becomes inactive and can be activated for a new communication.

The successful cancellation of a buffered send frees the buffer space occupied by the pending message.

Either the cancellation succeeds, or the communication succeeds, but not both. If a send is marked for cancellation, then it must be the case that either the send completes normally, in which case the message sent was received at the destination process, or that the send is successfully canceled, in which case no part of the message was received at the destination. Then, any matching receive has to be satisfied by another send. If a receive is marked for cancellation, then it must be the case that either the receive completes normally, or that the receive is successfully canceled, in which case no part of the receive buffer is altered. Then, any matching send has to be satisfied by another receive.

If the operation has been canceled, then information to that effect will be returned in the status argument of the operation that completes the communication.

MPI_TEST_CANCELLED(status, flag)
IN status	status object (Status)
OUT flag	(logical)

int MPI_Test_cancelled(MPI_Status *status, int *flag)

MPI_TEST_CANCELLED(STATUS, FLAG, IERROR)
LOGICAL FLAG
INTEGER STATUS(MPI_STATUS_SIZE), IERROR
bool MPI::Status::Is_cancelled() const

Returns flag = true if the communication associated with the status object was canceled successfully. In such a case, all other fields of status (such as count or tag) are undefined. Returns flag = false, otherwise. If a receive operation might be canceled then one should call MPI_TEST_CANCELLED first, to check whether the operation was canceled, before checking on the other fields of the return status.

Advice to users.

Cancel can be an expensive operation that should be used only exceptionally. ( End of advice to users.)

Advice to implementors.

If a send operation uses an ``eager'' protocol (data is transferred to the receiver before a matching receive is posted), then the cancellation of this send may require communication with the intended receiver in order to free allocated buffers. On some systems this may require an interrupt to the intended receiver. Note that, while communication may be needed to implement MPI_CANCEL, this is still a local operation, since its completion does not depend on the code executed by other processes. If processing is required on another process, this should be transparent to the application (hence the need for an interrupt and an interrupt handler). ( End of advice to implementors.)