8.6. Inter-Communication

This section introduces the concept of intercommunication and describes the portions of MPI that support it. It describes support for writing programs that contain user-level servers.

All communication described thus far has involved communication between MPI processes that are members of the same group. This type of communication is called `` intracommunication'' and the communicator used is called an `` intra-communicator,'' as we have noted earlier in the chapter.

In modular and multi-disciplinary applications, different MPI process groups execute distinct modules and MPI processes within different modules communicate with one another in a pipeline or a more general module graph. In these applications, the most natural way for a MPI process to specify a target MPI process is by the rank of the target MPI process within the target group. In applications that contain internal user-level servers, each server may be a MPI process group that provides services to one or more clients, and each client may be a MPI process group that uses the services of one or more servers. It is again most natural to specify the target MPI process by rank within the target group in these applications. This type of communication is called `` intercommunication'' and the communicator used is called an `` inter-communicator,'' as introduced earlier.

An inter-communication is a point-to-point communication between MPI processes in different groups. The group containing an MPI process that initiates an intercommunication operation is called the ``local group,'' that is, the sender in a send and the receiver in a receive. The group containing the target MPI process is called the ``remote group,'' that is, the receiver in a send and the sender in a receive. As in intracommunication, the target MPI process is specified using a ( communicator, rank) pair. Unlike intracommunication, the rank is relative to a second, remote group.

All inter-communicator constructors are blocking except for MPI_COMM_IDUP and require that the local and remote groups be disjoint.
Advice to users.

The groups must be disjoint for several reasons. First, the intent of the inter-communicators is to provide a communicator for communication between disjoint groups. This is reflected in the definition of MPI_INTERCOMM_MERGE, which allows the user to control the ranking of the MPI processes in the created intra-communicator; this ranking makes little sense if the groups are not disjoint. In addition, the natural extension of collective operations to inter-communicators makes the most sense when the groups are disjoint. ( End of advice to users.)
Here is a summary of the properties of intercommunication and inter-communicators:

• The syntax of point-to-point and collective communication is the same for both inter- and intracommunication. The same communicator can be used both for send and for receive operations.

• A target MPI process is addressed by its rank in the remote group, both for sends and for receives.

• Communications using an inter-communicator are guaranteed not to conflict with any communications that use a different communicator.

• A communicator will provide either intra- or intercommunication, never both.

Advice to implementors.

For the purpose of point-to-point communication, communicators can be represented in each process by a tuple consisting of: group
send_context
receive_context
source
For inter-communicators, group describes the remote group, and source is the rank of the MPI process in the local group. For intra-communicators, group is the communicator group (remote=local), source is the rank of the MPI process in this group, and send context and receive context are identical. A group can be represented by a rank-to-absolute-address translation table.

The inter-communicator cannot be discussed sensibly without considering MPI processes in both the local and remote groups. Imagine an MPI process P in group , which has an inter-communicator , and an MPI process Q in group , which has an inter-communicator . Then

• .group describes the group and .group describes the group .
• .send_context = .receive_context and the context is unique in ;
  .receive_context = .send_context and this context is unique in .
• .source is rank of P in and .source is rank of Q in .

Assume that Q posts a receive with an explicit source argument using the inter-communicator. Then Q matches receive_context to the message context and source argument to the message source.

The same algorithm is appropriate for intra-communicators as well.

In order to support inter-communicator accessors and constructors, it is necessary to supplement this model with additional structures, that store information about the local communication group, and additional safe contexts. ( End of advice to implementors.)