304. File Consistency

Consistency semantics define the outcome of multiple accesses to a single file. All file accesses in MPI are relative to a specific file handle created from a collective open. MPI provides three levels of consistency: sequential consistency among all accesses using a single file handle, sequential consistency among all accesses using file handles created from a single collective open with atomic mode enabled, and user-imposed consistency among accesses other than the above. Sequential consistency means the behavior of a set of operations will be as if the operations were performed in some serial order consistent with program order; each access appears atomic, although the exact ordering of accesses is unspecified. User-imposed consistency may be obtained using program order and calls to MPI_FILE_SYNC.

Let FH₁ be the set of file handles created from one particular collective open of the file FOO, and FH₂ be the set of file handles created from a different collective open of FOO. Note that nothing restrictive is said about FH₁ and FH₂: the sizes of FH₁ and FH₂ may be different, the groups of processes used for each open may or may not intersect, the file handles in FH₁ may be destroyed before those in FH₂ are created, etc. Consider the following three cases: a single file handle (e.g., ), two file handles created from a single collective open (e.g., and ), and two file handles from different collective opens (e.g., and ).

For the purpose of consistency semantics, a matched pair (Section Split Collective Data Access Routines ) of split collective data access operations (e.g., MPI_FILE_READ_ALL_BEGIN and MPI_FILE_READ_ALL_END) compose a single data access operation. Similarly, a nonblocking data access routine (e.g., MPI_FILE_IREAD) and the routine which completes the request (e.g., MPI_WAIT) also compose a single data access operation. For all cases below, these data access operations are subject to the same constraints as blocking data access operations.

Advice to users.

For an MPI_FILE_IREAD and MPI_WAIT pair, the operation begins when MPI_FILE_IREAD is called and ends when MPI_WAIT returns. ( End of advice to users.)
Assume that A₁ and A₂ are two data access operations. Let D₁ (D₂) be the set of absolute byte displacements of every byte accessed in A₁ (A₂). The two data accesses overlap if . The two data accesses conflict if they overlap and at least one is a write access.

Let SEQ_fh be a sequence of file operations on a single file handle, bracketed by MPI_FILE_SYNCs on that file handle. (Both opening and closing a file implicitly perform an MPI_FILE_SYNC.) SEQ_fh is a ``write sequence'' if any of the data access operations in the sequence are writes or if any of the file manipulation operations in the sequence change the state of the file (e.g., MPI_FILE_SET_SIZE or MPI_FILE_PREALLOCATE). Given two sequences, SEQ₁ and SEQ₂, we say they are not concurrent if one sequence is guaranteed to completely precede the other (temporally).

The requirements for guaranteeing sequential consistency among all accesses to a particular file are divided into the three cases given below. If any of these requirements are not met, then the value of all data in that file is implementation dependent.

Case 1: $fh_1 contained in FH_1$ All operations on fh₁ are sequentially consistent if atomic mode is set. If nonatomic mode is set, then all operations on fh₁ are sequentially consistent if they are either nonconcurrent, nonconflicting, or both.

Case 2: $fh_1a contained in FH_1$ and $fh_1b contained in FH_1$ Assume A₁ is a data access operation using fh_1a, and A₂ is a data access operation using fh_1b. If for any access A₁, there is no access A₂ that conflicts with A₁, then MPI guarantees sequential consistency.

However, unlike POSIX semantics, the default MPI semantics for conflicting accesses do not guarantee sequential consistency. If A₁ and A₂ conflict, sequential consistency can be guaranteed by either enabling atomic mode via the MPI_FILE_SET_ATOMICITY routine, or meeting the condition described in Case 3 below.

Case 3: $fh_1 contained in FH_1$ and $fh_2 contained in FH_2$ Consider access to a single file using file handles from distinct collective opens. In order to guarantee sequential consistency, MPI_FILE_SYNC must be used (both opening and closing a file implicitly perform an MPI_FILE_SYNC).

Sequential consistency is guaranteed among accesses to a single file if for any write sequence SEQ₁ to the file, there is no sequence SEQ₂ to the file which is concurrent with SEQ₁. To guarantee sequential consistency when there are write sequences, MPI_FILE_SYNC must be used together with a mechanism that guarantees nonconcurrency of the sequences.

See the examples in Section Examples for further clarification of some of these consistency semantics.

int MPI_File_set_atomicity(MPI_File fh, int flag)
MPI_File_set_atomicity(fh, flag, ierror)
TYPE(MPI_File), INTENT(IN) :: fh
LOGICAL, INTENT(IN) :: flag
INTEGER, OPTIONAL, INTENT(OUT) :: ierror
MPI_FILE_SET_ATOMICITY(FH, FLAG, IERROR)
INTEGER FH, IERROR
LOGICAL FLAG

Let FH be the set of file handles created by one collective open. The consistency semantics for data access operations using FH is set by collectively calling MPI_FILE_SET_ATOMICITY on FH. MPI_FILE_SET_ATOMICITY is collective; all processes in the group must pass identical values for fh and flag. If flag is true, atomic mode is set; if flag is false, nonatomic mode is set.

Changing the consistency semantics for an open file only affects new data accesses. All completed data accesses are guaranteed to abide by the consistency semantics in effect during their execution. Nonblocking data accesses and split collective operations that have not completed (e.g., via MPI_WAIT) are only guaranteed to abide by nonatomic mode consistency semantics.

Advice to implementors.

Since the semantics guaranteed by atomic mode are stronger than those guaranteed by nonatomic mode, an implementation is free to adhere to the more stringent atomic mode semantics for outstanding requests. ( End of advice to implementors.)

int MPI_File_get_atomicity(MPI_File fh, int *flag)
MPI_File_get_atomicity(fh, flag, ierror)
TYPE(MPI_File), INTENT(IN) :: fh
LOGICAL, INTENT(OUT) :: flag
INTEGER, OPTIONAL, INTENT(OUT) :: ierror
MPI_FILE_GET_ATOMICITY(FH, FLAG, IERROR)
INTEGER FH, IERROR
LOGICAL FLAG

MPI_FILE_GET_ATOMICITY returns the current consistency semantics for data access operations on the set of file handles created by one collective open. If flag is true, atomic mode is enabled; if flag is false, nonatomic mode is enabled.

int MPI_File_sync(MPI_File fh)
MPI_File_sync(fh, ierror)
TYPE(MPI_File), INTENT(IN) :: fh
INTEGER, OPTIONAL, INTENT(OUT) :: ierror
MPI_FILE_SYNC(FH, IERROR)
INTEGER FH, IERROR

Calling MPI_FILE_SYNC with fh causes all previous writes to fh by the calling process to be transferred to the storage device. If other processes have made updates to the storage device, then all such updates become visible to subsequent reads of fh by the calling process. MPI_FILE_SYNC may be necessary to ensure sequential consistency in certain cases (see above).

MPI_FILE_SYNC is a collective operation.

The user is responsible for ensuring that all nonblocking requests and split collective operations on fh have been completed before calling MPI_FILE_SYNC --- otherwise, the call to MPI_FILE_SYNC is erroneous.