16.3.8. Events

During the execution of an MPI application, the MPI implementation can raise events of a specific type to inform the user of a state change in the implementation. Event types describe specific state changes within the MPI implementation. In comparison to aggregate performance variables, events provide per-instance information on such state changes. The MPI implementation is said to raise an event when it invokes a callback function previously registered for the corresponding event type by the user. Each callback invocation for a specific event instance has a timestamp associated with it, which can be queried by the user, describing the time when the event was observed by the implementation. This decouples the observation of the state change from the communication of this information to the user. A timestamp in this context is a count of clock ticks elapsed since some time in the past and represented as a variable of type MPI_Count.

16.3.8.1. Event Sources

As a means to manage multiple state changes to be observed concurrently by different parts of the software and hardware system, the event interface of the MPI Tool Information Interface uses the concept of sources. A source in this context is a concept describing the logical entity raising the event. A source may or may not directly represent a concrete part of the software or hardware system. This concept is used primarily to describe partial ordering of events across different components where total ordering cannot necessarily be determined or is too costly to enforce.

The following function can be used to query the number of event sources, num_sources:

The number of available event sources can be queried with a call to MPI_T_SOURCE_GET_NUM. An MPI implementation is allowed to increase the number of sources during the execution of an MPI process. However, MPI implementations are not allowed to change the index of an event source or to delete an event source once it has been made visible to the user (e.g., if new event sources become available via dynamic loading of additional components in the MPI implementation).

A call to MPI_T_SOURCE_GET_INFO returns additional information on the source identified by the source_index argument.

The arguments name and name_len are used to return the name of the source as described in Section Convention for Returning Strings.

The arguments desc and desc_len are used to return the description of the source as described in Section Convention for Returning Strings.

The ordering argument returns whether event callbacks of this source will be invoked in chronological order, i.e., the timestamps reported by MPI_T_EVENT_GET_TIMESTAMP of subsequent events of the same source are monotonically increasing. The value of ordering can be MPI_T_SOURCE_ORDERED or MPI_T_SOURCE_UNORDERED.

The ticks_per_seconds argument returns the number of ticks elapsed in one second for the timer used for the specific source.

The max_ticks argument returns the largest number of ticks reported by this source as a timestamp before the value overflows.

Advice to users.

As the size of MPI_Count is defined in relation to the types MPI_Aint and MPI_Offset, the effective size of MPI_Count may lead to overflows of the timestamp values reported. Users can use the argument max_ticks to mitigate resulting problems. ( End of advice to users.)
MPI can optionally return an info object containing the default hints set for this source. If the argument to info provided by the user is the NULL pointer, this argument is ignored, otherwise an MPI implementation is required to return all hints that are supported by the implementation for this source and have default values specified; any user-supplied hints that were not ignored by the implementation; and any additional hints that were set by the implementation. If no such hints exist, a handle to a newly created info object is returned that contains no key/value pair. The user is responsible for freeing info via MPI_INFO_FREE.

To enable proper query of a reference timestamp for a specific source, a user can obtain a current timestamp using MPI_T_SOURCE_GET_TIMESTAMP. The argument source_index identifies the index of the source to query. The call returns MPI_SUCCESS and a current timestamp in the argument timestamp if the source supports ad-hoc generation of timestamps. The call returns MPI_T_ERR_INVALID_INDEX if the index does not identify a valid source. The call returns MPI_T_ERR_NOT_SUPPORTED if the source does not support the ad-hoc generation of timestamps.

16.3.8.2. Callback Safety Requirements

The actions a user is allowed to perform inside a callback function may vary with its execution context. As the user has no control over the execution context of specific callback function invocations, MPI provides a way to communicate this information using callback safety levels.

Table 21: Hierarchy of safety requirement levels for event callback routines

The level of MPI_T_CB_REQUIRE_NONE is the lowest level and does not impose any restrictions on the callback function.

The level of MPI_T_CB_REQUIRE_MPI_RESTRICTED restricts the set of MPI functions that can be called from inside the callback to all functions with the prefix MPI_T as well as MPI_WTICK and MPI_WTIME.

Advice to users.

While some MPI functions are safe to be called inside a callback function used in the MPI tool information interface---which may in some implementations be issued from asynchronous contexts such as signal handlers---this does not imply that those MPI functions are generally safe to be called in asynchronous contexts such as signal handlers. ( End of advice to users.)
The level of MPI_T_CB_REQUIRE_THREAD_SAFE includes all the limitations of MPI_T_CB_REQUIRE_MPI_RESTRICTED and additionally requires the callback to be reentrant and thread-safe. This means the callback must allow its execution to be interrupted by or happen concurrently with any other callback including itself.

The level of MPI_T_CB_REQUIRE_ASYNC_SIGNAL_SAFE includes all the limitations of MPI_T_CB_REQUIRE_THREAD_SAFE and additionally requires the callback to meet the safety requirements needed to support invocations from asynchronous contexts, such as signal handlers.

Advice to users.

It is always safe to assume the highest restrictions for a callback invocation (i.e., MPI_T_CB_REQUIRE_ASYNC_SIGNAL_SAFE). By evaluating the specific requirements at runtime, a tool may obtain more freedom of action within the callback. ( End of advice to users.)

Advice to implementors.

A high-quality implementation will strive to set callback safety requirements to the most permissive level for a given callback invocation. ( End of advice to implementors.)
All functions with the prefix MPI_T, except those listed in Table 22, may return the return code MPI_T_ERR_NOT_ACCESSIBLE to indicate that the user may not access this function at this time. The functions (and their respective PMPI versions) listed in Table 22 are exceptions to this rule and shall not return MPI_T_ERR_NOT_ACCESSIBLE.

Table 22: List of MPI functions that when called from within a callback function may not return MPI_T_ERR_NOT_ACCESSIBLE

A call may be implemented in a way that is not safe for all execution contexts of a callback function, e.g., inside a signal handler. An MPI implementation therefore needs a way to communicate its inability to perform a certain action due to the execution context of a callback invocation. ( End of rationale.)

Advice to implementors.

A high-quality implementation shall not return MPI_T_ERR_NOT_ACCESSIBLE except where absolutely necessary. ( End of advice to implementors.)

Advice to users.

Users intercepting calls into the MPI tool information interface using the PMPI interface must ensure that the safety requirements for the calling context are met. This means that users may have to implement the wrapper with the highest safety level used by the MPI implementation. ( End of advice to users.)

16.3.8.3. Event Type Query Functions

An MPI implementation exports a set of N event types through the MPI tool information interface. If N is zero, then the MPI implementation does not export any event types; otherwise, the provided event types are indexed from 0 to N-1. This index number is used in subsequent calls to identify a specific event type.

An MPI implementation is allowed to increase the number of event types during the execution of an MPI process. However, MPI implementations are not allowed to change the index of an event type or to delete an event type once it has been made visible to the user (e.g., if new event types become available via dynamic loading of additional components in the MPI implementation).

The following function can be used to query the number of event types, num_events:

The function MPI_T_EVENT_GET_INFO provides access to additional information about a specific event type.

After a successful call to MPI_T_EVENT_GET_INFO for a particular event type, subsequent calls to this routine that query information about the same event type must return the same information. If any INOUT or OUT argument to MPI_T_EVENT_GET_INFO is a NULL pointer, the implementation will ignore the argument and not return a value for the specific argument.

The arguments name and name_len are used to return the name of the event type as described in Section Convention for Returning Strings. If completed successfully, the routine is required to return a name of at least length one. The name of the event type must be unique with respect to all other names for event types used by the MPI implementation.

The argument verbosity returns the verbosity level of the event type (see Section Verbosity Levels).

The argument array_of_datatypes returns an array of MPI datatype handles that describe the elements returned for an instance of the event type with index event_index. The event data can either be queried element by element with MPI_T_EVENT_READ or copied into a contiguous event buffer with MPI_T_EVENT_COPY. For the latter case, the argument array_of_displacements returns an array of byte displacements in the event buffer in ascending order starting with zero.

The user is responsible for the memory allocation for the array_of_datatypes and array_of_displacements arrays. The number of elements in each array is supplied by the user in num_elements. If the number of elements used by the event type is larger than the value of num_elements provided by the user, the number of datatype handles and displacements returned in the corresponding arrays is truncated to the value of num_elements passed in by the user. If the user passes the NULL pointer for array_of_datatypes or array_of_displacements, the respective arguments are ignored. Unless the user passes the NULL pointer for num_elements, the function returns the number of elements required for this event type. If the user passes the NULL pointer for num_elements, the arguments num_elements, array_of_datatypes, and array_of_displacements are ignored.

MPI can optionally return an enumeration identifier in the enumtype argument, describing the individual elements in the array_of_datatypes argument. Otherwise, enumtype is set to MPI_T_ENUM_NULL. If the argument to enumtype provided by the user is the NULL pointer, no enumeration type is returned.

MPI can optionally return an info object containing the default hints set for a registration handle for this event type. If the argument to info provided by the user is the NULL pointer, this argument is ignored, otherwise an MPI implementation is required to return all hints that are supported by the implementation for a registration handle for this event type and have default values specified; any user-supplied hints that were not ignored by the implementation; and any additional hints that were set by the implementation. If no such hints exist, a handle to a newly created info object is returned that contains no key/value pair. The user is responsible for freeing info via MPI_INFO_FREE.

The arguments desc and desc_len are used to return the description of the event type as described in Section Convention for Returning Strings. Returning a description is optional. If an MPI implementation does not return a description, the first character for desc must be set to the null character and desc_len must be set to one at the return from this function.

The parameter bind returns the type of the MPI object to which the event type must be bound or the value MPI_T_BIND_NO_OBJECT (see Section Binding MPI Tool Information Interface Variables to MPI Objects).

If an event type has an equivalent name across connected MPI processes, the following OUT parameters must be identical: verbosity, array_of_datatypes, num_elements, enumtype, and bind. The returned description must be equivalent. As the argument array_of_displacements is process dependent, it may differ across connected MPI processes.

This routine returns MPI_SUCCESS on success and returns MPI_T_ERR_INVALID_INDEX if event_index does not match a valid event type index provided by the implementation at the time of the call.

MPI_T_EVENT_GET_INDEX returns the index of an event type identified by a known event type name. The name parameter is provided by the caller, and event_index is returned by the MPI implementation. The name parameter is a string terminated with a null character.

This routine returns MPI_SUCCESS on success and returns MPI_T_ERR_INVALID_NAME if name does not match the name of any event type provided by the implementation at the time of the call.

Rationale.

This routine is provided to enable fast retrieval of an event index by a tool, assuming it knows the name of the event type for which it is looking. The number of event types exposed by the implementation can change over time, so it is not possible for the tool to simply iterate over the list of event types once at initialization. Although using MPI implementation specific event type names is not portable across MPI implementations, tool developers may choose to take this route for lower overhead at runtime because the tool will not have to iterate over the entire set of event types to find a specific one. ( End of rationale.)

16.3.8.4. Handle Allocation and Deallocation

Before the MPI implementation calls a callback function on the occurrence of a specific event, the user needs to register a callback function to be called for that event type and obtain a handle of type MPI_T_event_registration.

MPI_T_EVENT_HANDLE_ALLOC creates a registration handle for the event type identified by event_index. Furthermore, if required by the event type, the registration handle is bound to the object referred to by the argument obj_handle. The argument obj_handle is ignored if the MPI_T_EVENT_GET_INFO call for this event type returned MPI_T_BIND_NO_OBJECT in the argument bind. The user can pass hints for the handle allocation to the MPI implementation via the info argument. The allocated event-registration handle is returned in the argument event_registration.

MPI_T_EVENT_HANDLE_SET_INFO updates the hints of the event-registration handle associated with event_registration using the hints provided in info. A call to this procedure has no effect on previously set or defaulted hints that are not specified by info. It also has no effect on previously set or defaulted hints that are specified by info, but are ignored by the MPI implementation in this call to MPI_T_EVENT_HANDLE_SET_INFO.

Advice to users.

Some info items that an implementation can use when it creates an event-registration handle cannot easily be changed once the registration handle is created. Thus, an implementation may ignore hints issued in this call that it would have accepted in a handle allocation call. An implementation may also be unable to update certain info hints in a call to MPI_T_EVENT_HANDLE_SET_INFO. MPI_T_EVENT_HANDLE_GET_INFO can be used to determine whether info changes were ignored by the implementation. ( End of advice to users.)

MPI_T_EVENT_HANDLE_GET_INFO returns a new info object containing the hints of the event-registration handle associated with event_registration. The current setting of all hints related to this registration handle is returned in info_used. An MPI implementation is required to return all hints that are supported by the implementation and have default values specified; any user-supplied hints that were not ignored by the implementation; and any additional hints that were set by the implementation. If no such hints exist, a handle to a newly created info object is returned that contains no key/value pairs. The user is responsible for freeing info_used via MPI_INFO_FREE.

MPI_T_EVENT_REGISTER_CALLBACK associates a user-defined function pointed to by event_cb_function with an allocated event-registration handle. The maximum callback safety level supported by the callback function is passed in the argument cb_safety. The safety levels are defined in Table 21. A user can register multiple callback functions for a given event-registration handle, potentially specifying one for each callback safety level. Registering a callback function for a specific callback safety level overwrites any previously-registered callback function pointer and info object associated with the event registration for the specific callback safety level. If event_cb_function is the NULL pointer, an existing association of a callback function for that callback safety level is removed.

When an event is triggered, the implementation will select from all registered callbacks the callback with the lowest safety level valid in the context in which the callback is invoked. In situations where the required callback safety level exceeds the highest level for which a callback function is registered for a given registration handle, the event instance is dropped.

At callback invocation time, the implementation passes the pointer to a user-defined memory region specified during callback registration with the argument user_data.

The user can pass hints for the registration of the specified callback function to the MPI implementation via the info argument.

Advice to users.

As event instances can be raised as soon as the registration handle is associated with the first callback function, the callback function with the highest callback safety guarantees should be registered before any further registrations for lower callback safety guarantees, to avoid dropped events due to insufficient callback safety guarantees. ( End of advice to users.)
The callback function passed to MPI_T_EVENT_REGISTER_CALLBACK in the argument event_cb_function needs to have the following type:

typedef void MPI_T_event_cb_function(MPI_T_event_instance event_instance, MPI_T_event_registration event_registration, MPI_T_cb_safety cb_safety, void *user_data);

The argument event_instance corresponds to a handle for the opaque event-instance object of type MPI_T_event_instance. This handle is only valid inside the corresponding invocation of the function to which it is passed. The argument event_registration corresponds to the event-registration handle returned by MPI_T_EVENT_HANDLE_ALLOC for the user function to the same event type and bound object combination. The handle can be used to identify the specific event registration information, such as event type and bound object, or even to deallocate the handle from within the callback invocation. The argument cb_safety describes the safety requirements the callback function must fulfill in the current invocation. The argument user_data is the pointer to user-allocated memory that was passed to the MPI implementation during callback registration.

MPI_T_EVENT_CALLBACK_SET_INFO updates the hints of the callback function registered for the callback safety level specified by cb_safety of the event-registration handle associated with event_registration using the hints provided in info. A call to this procedure has no effect on previously set or defaulted hints that are not specified by info. It also has no effect on previously set or defaulted hints that are specified by info, but are ignored by the MPI implementation in this call to MPI_T_EVENT_CALLBACK_SET_INFO.

MPI_T_EVENT_CALLBACK_GET_INFO returns a new info object containing the hints of the callback function registered for the callback safety level specified by cb_safety of the event-registration handle associated with event_registration. The current set of all hints related to this callback safety level of the event-registration handle is returned in info_used. An MPI implementation is required to return all hints that are supported by the implementation and have default values specified, any user-supplied hints that were not ignored by the implementation, and any additional hints that were set by the implementation. If no such hints exist, a handle to a newly created info object is returned that contains no key/value pairs. The user is responsible for freeing info_used via MPI_INFO_FREE.

To stop the MPI implementation from raising events for a specific registration, a user needs to free the corresponding event-registration handle.

MPI_T_EVENT_HANDLE_FREE returns MPI_SUCCESS when deallocation of the handle was initiated successfully and returns MPI_T_ERR_INVALID_HANDLE if event_registration does not match a valid allocated event-registration handle at the time of the call. The callback function free_cb_function is called by the MPI implementation, when it is able to guarantee that no further event instances for the corresponding event-registration handle will be raised. If the pointer to free_cb_function is the NULL pointer, no user function is invoked after successful deallocation of the event registration handle. The pointer to user-controlled memory provided in the user_data argument will be passed to the function provided in the free_cb_function on invocation.

Advice to users.

A free-callback function associated with a registration handle should always be prepared to postpone any pending actions, should the provided callback safety requirements exceed those required by the pending actions. ( End of advice to users.)
The callback function passed to MPI_T_EVENT_HANDLE_FREE in the argument free_cb_function needs to have the following type:

typedef void MPI_T_event_free_cb_function(MPI_T_event_registration event_registration, MPI_T_cb_safety cb_safety, void *user_data);

16.3.8.5. Handling Dropped Events

Events may occur at times when the MPI implementation cannot invoke the user function corresponding to a matching event handle. An implementation is allowed to buffer such events and delay the callback invocation. If an event occurs at times when the corresponding callback function cannot be called and the corresponding data cannot be buffered, or no callback function meeting the required callback safety level is registered, the event data may be dropped. To discover such data loss, the user can set a handler function for a specific event-registration handle.

MPI_T_EVENT_SET_DROPPED_HANDLER registers the function dropped_cb_function to be called by the MPI implementation when event information is dropped for the registration handle specified in event_registration. Subsequent calls to MPI_T_EVENT_SET_DROPPED_HANDLER with the same registration handle will replace previously-registered callback functions for that registration handle. If the pointer to dropped_cb_function is the NULL pointer, no data loss is recorded or reported until a new valid callback function is registered.

Advice to users.

The invocation of the dropped handler callback function may not necessarily occur close to the time the event was actually lost. ( End of advice to users.)
The callback function passed to MPI_T_EVENT_SET_DROPPED_HANDLER in the argument dropped_cb_function needs to have the following type:

typedef void MPI_T_event_dropped_cb_function(MPI_Count count, MPI_T_event_registration event_registration, int source_index, MPI_T_cb_safety cb_safety, void *user_data);

The argument event_registration corresponds to the event registration handle to which the dropped data corresponds. The argument count provides a best effort estimation of the number of invocations to a registered event callback corresponding to event_registration that were not executed since the registration of the dropped-callback handler or the last invocation of a registered dropped-callback handler. If the number of dropped events observed by the implementation exceeds the limit of count, an implementation shall set count to the maximum possible value for the type of count. The source_index provides the index of the source that dropped the corresponding event information. The argument cb_safety describes the safety requirements the callback function must fulfill in the current invocation. The possible values for cb_safety are described in Table 21. The argument user_data is the pointer to user-allocated memory that was passed to the MPI implementation during callback registration. If no event callback is registered for safety requirement levels that an implementation uses to invoke the dropped handler callback function for a specific event, the corresponding dropped handler callback function will not be invoked.

Advice to users.

A callback function for dropped events associated with a registration handle should always be prepared to postpone any pending actions, should the provided callback safety requirements exceed those required by the pending actions. ( End of advice to users.)

Advice to implementors.

A high-quality implementation should strive to find a good balance between timely notification, completeness of information, and the freedom of action for a tool when invoking the callback function for dropped events associated with a registration handle. ( End of advice to implementors.)
If dropped event notifications have been observed for a specific source since the last event notification of that source, the corresponding dropped handler callback function must be called before other events are raised for that source. This means in a sequence of five events E1 to E5 from the same source, where E3 and E4 were dropped, any handler function set through MPI_T_EVENT_SET_DROPPED_HANDLER for event-registration handles associated with E3 or E4 must be called before E5 is raised.

16.3.8.6. Reading Event Data

In event callbacks, the parameter event_instance provides access to the per-instance event data, i.e., the data encoded by the specific event type for this instance. The user can obtain event data as well as event meta data, such as a time stamp and the source, by providing this handle to the respective query functions. The event-instance handle is invalid beyond the scope of the current invocation of the callback function to which it is provided.

The callback function argument event_registration identifies the registration handle that was used to register the callback function.

The callback function argument cb_safety indicates the requirements for the specific callback invocation. The value is one of the safety requirements levels described in Table 21. The argument user_data passes the pointer provided by the user during callback registration back to the function call.

Advice to users.

Depending on the registered event and usage of MPI by the application, a callback function may be invoked with high frequency. Users should therefore strive to minimize the amount of work done inside callback functions. Furthermore, the time spent in a callback function may influence the capability of an implementation to buffer events; long execution times may lead to an increased number of dropped events. ( End of advice to users.)
MPI provides the following function calls to access data of a specific event instance and its corresponding meta data (such as its time and source).

MPI_T_EVENT_READ allows users to copy one element of the event data to a user-specified buffer at a time.

The event_instance argument identifies the event instance to query. It is erroneous to provide any other event-instance handle to the call than the one passed by the MPI implementation to the callback function in which the data is read. The buffer argument must point to a memory location the MPI implementation can copy the element of the event data to identified by element_index.

MPI_T_EVENT_COPY copies the event data as a whole into the user-provided buffer. The user must assure that the buffer is of at least the size of the extent of the event type, which can be computed from the type and displacement information returned by the corresponding call to MPI_T_EVENT_GET_INFO. The data may include padding bytes between individual elements of the event data in the buffer. A user can reconstruct the location and size of the data contained in the buffer through the information returned by MPI_T_EVENT_GET_INFO.

Advice to implementors.

An implementation should strive to use an appropriately compact representation when copying event instance data to a user buffer via MPI_T_EVENT_COPY to reduce the amount of memory required for the user buffer. ( End of advice to implementors.)

16.3.8.7. Reading Event Meta Data

Additional to the specific event data encoded by each event type, supplemental information available across all event types can be queried.

MPI_T_EVENT_GET_TIMESTAMP returns the timestamp of when the event was initially observed by the implementation. The event_instance argument identifies the event instance to query. It is erroneous to provide any other handle to the call than the one passed by the MPI implementation to the callback function in which the timestamp is read.

Advice to users.

An MPI implementation may postpone the call to the user's callback function. In this case, the call to MPI_T_EVENT_GET_TIMESTAMP may yield a timestamp in the past that is closer to the time the event was initially observed, as opposed to a timestamp captured during callback function invocation. ( End of advice to users.)

Advice to implementors.

A high-quality implementation will return a timestamp as close as possible to the earliest time the event was observed by the MPI implementation. ( End of advice to implementors.)
An event may be raised from different components acting as event sources in the MPI implementation. A source in this context is an abstract concept that helps to define partial ordering of raised events, as each source provides its own ordering guarantees. A source describes the entity that raises the event, rather than the origin of the data.

To identify the source of an event instance, the user can query the index of the source within the corresponding event callback function invocation.

Advice to implementors.

An excessive number of event sources may negatively impact performance of a tool due to per-source overhead in event handling. ( End of advice to implementors.)

The event_instance argument identifies the event instance to query. It is erroneous to provide any other event-instance handle to the call than the one passed by the MPI implementation to the callback function in which the source is queried.

The source_index argument returns the index of the source of the event instance. It can be used to query more information on the source using MPI_T_SOURCE_GET_INFO.

Rationale.

Event callback function invocations are associated with a source to enable chronological processing of events on the tool side, when required, while retaining low overhead on the side of the MPI implementation. ( End of rationale.)