258. Split Collective Data Access Routines

MPI provides a restricted form of ``nonblocking collective'' I/O operations for all data accesses using split collective data access routines. These routines are referred to as ``split'' collective routines because a single collective operation is split in two: a begin routine and an end routine. The begin routine begins the operation, much like a nonblocking data access (e.g., MPI_FILE_IREAD). The end routine completes the operation, much like the matching test or wait (e.g., MPI_WAIT). As with nonblocking data access operations, the user must not use the buffer passed to a begin routine while the routine is outstanding; the operation must be completed with an end routine before it is safe to free buffers, etc.

Split collective data access operations on a file handle fh are subject to the semantic rules given below.

• On any MPI process, each file handle may have at most one active split collective operation at any time.

• Begin calls are collective over the group of processes that participated in the collective open and follow the ordering rules for collective calls.

• End calls are collective over the group of processes that participated in the collective open and follow the ordering rules for collective calls. Each end call matches the preceding begin call for the same collective operation. When an ``end'' call is made, exactly one unmatched ``begin'' call for the same operation must precede it.

• An implementation is free to implement any split collective data access routine using the corresponding blocking collective routine when either the begin call (e.g., MPI_FILE_READ_ALL_BEGIN) or the end call (e.g., MPI_FILE_READ_ALL_END) is issued. The begin and end calls are provided to allow the user and MPI implementation to optimize the collective operation.

• Split collective operations do not match the corresponding regular collective operation. For example, in a single collective read operation, an MPI_FILE_READ_ALL on one process does not match an MPI_FILE_READ_ALL_BEGIN/ MPI_FILE_READ_ALL_END pair on another process.

• Split collective routines must specify a buffer in both the begin and end routines. By specifying the buffer that receives data in the end routine, we can avoid many (though not all) of the problems described in ``A Problem with Register Optimization,'' Section A Problem with Register Optimization , page A Problem with Register Optimization .

• No collective I/O operations are permitted on a file handle concurrently with a split collective access on that file handle (i.e., between the begin and end of the access). That is
  

                  MPI_File_read_all_begin(fh, ...); 
                  ... 
                  MPI_File_read_all(fh, ...); 
                  ... 
                  MPI_File_read_all_end(fh, ...); 
  

  is erroneous.

• In a multithreaded implementation, any split collective begin and end operation called by a process must be called from the same thread. This restriction is made to simplify the implementation in the multithreaded case. (Note that we have already disallowed having two threads begin a split collective operation on the same file handle since only one split collective operation can be active on a file handle at any time.)

For the purpose of consistency semantics (Section File Consistency , page File Consistency ), a matched pair of split collective data access operations (e.g., MPI_FILE_READ_ALL_BEGIN and MPI_FILE_READ_ALL_END) compose a single data access.

MPI_FILE_READ_AT_ALL_BEGIN(fh, offset, buf, count, datatype)
IN fh	file handle (handle)
IN offset	file offset (integer)
OUT buf	initial address of buffer (choice)
IN count	number of elements in buffer (integer)
IN datatype	datatype of each buffer element (handle)

int MPI_File_read_at_all_begin(MPI_File fh, MPI_Offset offset, void *buf, int count, MPI_Datatype datatype)
MPI_FILE_READ_AT_ALL_BEGIN(FH, OFFSET, BUF, COUNT, DATATYPE, IERROR)
<type> BUF(*)
INTEGER FH, COUNT, DATATYPE, IERROR
INTEGER(KIND=MPI_OFFSET_KIND) OFFSET
void MPI::File::Read_at_all_begin(MPI::Offset offset, void* buf, int count, const MPI::Datatype& datatype)

MPI_FILE_READ_AT_ALL_END(fh, buf, status)
IN fh	file handle (handle)
OUT buf	initial address of buffer (choice)
OUT status	status object (Status)

int MPI_File_read_at_all_end(MPI_File fh, void *buf, MPI_Status *status)
MPI_FILE_READ_AT_ALL_END(FH, BUF, STATUS, IERROR)
<type> BUF(*)
INTEGER FH, STATUS(MPI_STATUS_SIZE), IERROR
void MPI::File::Read_at_all_end(void* buf, MPI::Status& status)
void MPI::File::Read_at_all_end(void* buf)

MPI_FILE_WRITE_AT_ALL_BEGIN(fh, offset, buf, count, datatype)
INOUT fh	file handle (handle)
IN offset	file offset (integer)
IN buf	initial address of buffer (choice)
IN count	number of elements in buffer (integer)
IN datatype	datatype of each buffer element (handle)

int MPI_File_write_at_all_begin(MPI_File fh, MPI_Offset offset, void *buf, int count, MPI_Datatype datatype)
MPI_FILE_WRITE_AT_ALL_BEGIN(FH, OFFSET, BUF, COUNT, DATATYPE, IERROR)
<type> BUF(*)
INTEGER FH, COUNT, DATATYPE, IERROR
INTEGER(KIND=MPI_OFFSET_KIND) OFFSET
void MPI::File::Write_at_all_begin(MPI::Offset offset, const void* buf, int count, const MPI::Datatype& datatype)

MPI_FILE_WRITE_AT_ALL_END(fh, buf, status)
INOUT fh	file handle (handle)
IN buf	initial address of buffer (choice)
OUT status	status object (Status)

int MPI_File_write_at_all_end(MPI_File fh, void *buf, MPI_Status *status)
MPI_FILE_WRITE_AT_ALL_END(FH, BUF, STATUS, IERROR)
<type> BUF(*)
INTEGER FH, STATUS(MPI_STATUS_SIZE), IERROR
void MPI::File::Write_at_all_end(const void* buf, MPI::Status& status)
void MPI::File::Write_at_all_end(const void* buf)

MPI_FILE_READ_ALL_BEGIN(fh, buf, count, datatype)
INOUT fh	file handle (handle)
OUT buf	initial address of buffer (choice)
IN count	number of elements in buffer (integer)
IN datatype	datatype of each buffer element (handle)

int MPI_File_read_all_begin(MPI_File fh, void *buf, int count, MPI_Datatype datatype)
MPI_FILE_READ_ALL_BEGIN(FH, BUF, COUNT, DATATYPE, IERROR)
<type> BUF(*)
INTEGER FH, COUNT, DATATYPE, IERROR
void MPI::File::Read_all_begin(void* buf, int count, const MPI::Datatype& datatype)

MPI_FILE_READ_ALL_END(fh, buf, status)
INOUT fh	file handle (handle)
OUT buf	initial address of buffer (choice)
OUT status	status object (Status)

int MPI_File_read_all_end(MPI_File fh, void *buf, MPI_Status *status)
MPI_FILE_READ_ALL_END(FH, BUF, STATUS, IERROR)
<type> BUF(*)
INTEGER FH, STATUS(MPI_STATUS_SIZE), IERROR
void MPI::File::Read_all_end(void* buf, MPI::Status& status)
void MPI::File::Read_all_end(void* buf)

MPI_FILE_WRITE_ALL_BEGIN(fh, buf, count, datatype)
INOUT fh	file handle (handle)
IN buf	initial address of buffer (choice)
IN count	number of elements in buffer (integer)
IN datatype	datatype of each buffer element (handle)

int MPI_File_write_all_begin(MPI_File fh, void *buf, int count, MPI_Datatype datatype)
MPI_FILE_WRITE_ALL_BEGIN(FH, BUF, COUNT, DATATYPE, IERROR)
<type> BUF(*)
INTEGER FH, COUNT, DATATYPE, IERROR
void MPI::File::Write_all_begin(const void* buf, int count, const MPI::Datatype& datatype)

MPI_FILE_WRITE_ALL_END(fh, buf, status)
INOUT fh	file handle (handle)
IN buf	initial address of buffer (choice)
OUT status	status object (Status)

int MPI_File_write_all_end(MPI_File fh, void *buf, MPI_Status *status)
MPI_FILE_WRITE_ALL_END(FH, BUF, STATUS, IERROR)
<type> BUF(*)
INTEGER FH, STATUS(MPI_STATUS_SIZE), IERROR
void MPI::File::Write_all_end(const void* buf, MPI::Status& status)
void MPI::File::Write_all_end(const void* buf)

MPI_FILE_READ_ORDERED_BEGIN(fh, buf, count, datatype)
INOUT fh	file handle (handle)
OUT buf	initial address of buffer (choice)
IN count	number of elements in buffer (integer)
IN datatype	datatype of each buffer element (handle)

int MPI_File_read_ordered_begin(MPI_File fh, void *buf, int count, MPI_Datatype datatype)
MPI_FILE_READ_ORDERED_BEGIN(FH, BUF, COUNT, DATATYPE, IERROR)
<type> BUF(*)
INTEGER FH, COUNT, DATATYPE, IERROR
void MPI::File::Read_ordered_begin(void* buf, int count, const MPI::Datatype& datatype)

MPI_FILE_READ_ORDERED_END(fh, buf, status)
INOUT fh	file handle (handle)
OUT buf	initial address of buffer (choice)
OUT status	status object (Status)

int MPI_File_read_ordered_end(MPI_File fh, void *buf, MPI_Status *status)
MPI_FILE_READ_ORDERED_END(FH, BUF, STATUS, IERROR)
<type> BUF(*)
INTEGER FH, STATUS(MPI_STATUS_SIZE), IERROR
void MPI::File::Read_ordered_end(void* buf, MPI::Status& status)
void MPI::File::Read_ordered_end(void* buf)

MPI_FILE_WRITE_ORDERED_BEGIN(fh, buf, count, datatype)
INOUT fh	file handle (handle)
IN buf	initial address of buffer (choice)
IN count	number of elements in buffer (integer)
IN datatype	datatype of each buffer element (handle)

int MPI_File_write_ordered_begin(MPI_File fh, void *buf, int count, MPI_Datatype datatype)
MPI_FILE_WRITE_ORDERED_BEGIN(FH, BUF, COUNT, DATATYPE, IERROR)
<type> BUF(*)
INTEGER FH, COUNT, DATATYPE, IERROR
void MPI::File::Write_ordered_begin(const void* buf, int count, const MPI::Datatype& datatype)

MPI_FILE_WRITE_ORDERED_END(fh, buf, status)
INOUT fh	file handle (handle)
IN buf	initial address of buffer (choice)
OUT status	status object (Status)

int MPI_File_write_ordered_end(MPI_File fh, void *buf, MPI_Status *status)
MPI_FILE_WRITE_ORDERED_END(FH, BUF, STATUS, IERROR)
<type> BUF(*)
INTEGER FH, STATUS(MPI_STATUS_SIZE), IERROR
void MPI::File::Write_ordered_end(const void* buf, MPI::Status& status)
void MPI::File::Write_ordered_end(const void* buf)