3.4.2. MPI Procedures

All MPI procedures can either be local or nonlocal---defined as follows:

Nonlocal procedure:

An MPI procedure is nonlocal if returning may require, during its execution, some specific semantically-related MPI procedure to be called on another MPI process.

Local procedure:

An MPI procedure is local if it is not nonlocal.

There are also other MPI procedures that do not implement any stage of any MPI operation.

The semantics of MPI operation-related procedures are described using two orthogonal (independent) concepts: completeness (depends on which stages are included) and locality. Such procedures can be either incomplete, or completing, or freeing, or completing and freeing based on the status of the associated operation at the time the procedure returns. Also, all such procedures can be described as either blocking or nonblocking, but these latter two terms refer to combinations of the completeness and locality concepts. Additionally, all MPI operation-related procedures can be collective or noncollective.

The following are properties of MPI operation-related procedures:

Initialization procedure:

An MPI procedure is an initialization procedure if return from the procedure indicates that the associated operation has completed its initialization stage, which implies that the user has handed over control of the argument list (but not contents of the data buffers) to MPI. The user is still allowed to read or modify the contents of the data buffers. If an initializing procedure is not also the freeing procedure of the associated operation (see below) then the user is not permitted to deallocate the data buffers or to modify the array arguments.

Starting procedure:

An MPI procedure is a starting procedure if return from the procedure indicates that the associated operation has completed its starting stage, which implies that the user has handed over control of the data buffers to MPI. If a starting procedure is not also a completing procedure of the associated operation (see below) then the user is not permitted to modify input data buffers or to read output data buffers.

Initiation procedure:

An MPI procedure is an initiation procedure if return from the procedure indicates that both the initialization and the starting stage have completed, which implies control of the entire argument list is handed over to MPI.

Completing procedure:

An MPI procedure is called completing if return from the procedure indicates that at least one associated operation has finished its completion stage, which implies that the user can rely on the content of the output data buffers and modify the content of input and output data buffers of such operation(s). If a completing procedure is not also a freeing procedure (see below) then the user is not permitted to deallocate the data buffers or to modify the array arguments.

Incomplete procedure:

An MPI procedure is called incomplete if it is not a completing procedure.

Freeing procedure:

An MPI procedure is freeing if return from the procedure indicates that at least one associated operation has finished its freeing stage, which implies that the user can reuse all parameters specified when initializing such associated operation(s).

Nonblocking procedure:

An MPI procedure is nonblocking if it is incomplete and local.

Blocking procedure:

An MPI procedure is blocking if it is not nonblocking.

Note that for operation-related MPI procedures, in most cases incomplete procedures are local and completing procedures are nonlocal. Exceptions are noted where such procedures are defined. In many cases an additional prefix letter I as an abbreviation of the words incomplete and immediate marks nonblocking procedures in the procedure name.

Some categorization examples are listed below.

Nonblocking procedures:

• incomplete and local: MPI_ISEND, MPI_IRECV, MPI_IBCAST, MPI_IMPROBE, MPI_SEND_INIT, MPI_RECV_INIT, ...

• completing and nonlocal: MPI_SEND, MPI_RECV, MPI_BCAST, ...
• incomplete and nonlocal: MPI_MPROBE, MPI_BCAST_INIT, ..., MPI_FILE_{READ|WRITE}_{AT_ALL|ALL|ORDERED}_BEGIN.
• completing and local: MPI_BSEND, MPI_RSEND, MPI_MRECV.

• MPI_COMM_RANK, MPI_WTIME, MPI_PROBE, MPI_IPROBE, ...

Collective procedure:

An MPI procedure is collective if all processes in a group or groups of MPI processes need to invoke the procedure.

Initialization procedures of collective operations over the same process group must be executed in the same order by all members of the process group.

An MPI collective procedure is synchronizing if it will only return once all processes in the associated group or groups of MPI processes have called the appropriate matching MPI procedure.

The initiation procedures for nonblocking collective operations and the starting procedures for persistent collective operations are local and shall not be synchronizing.

All other procedures for collective operations, such as for blocking collective operations and the initialization procedures for persistent collective operations, may or may not be synchronizing.

Advice to users.

Calling any synchronizing function is erroneous when there is no possibility of corresponding calls at all other processes in the associated process group.

Waiting for completion of any collective operation is erroneous when there is no possibility that all other processes in the associated group will be able to start the corresponding operation. ( End of advice to users.)

Noncollective procedure:

Noncollective procedures are defined as procedures that are not collective.