MPI support functionality. More...

Classes
class	Comm
	A duplicate MPI communicator and manage lifetime of the communicator. More...

Enumerations
enum class	tag : int { consensus_pcx = 1200 , consensus_nbx = 1202 }
	MPI communication tags.

Functions
int	rank (MPI_Comm comm)
	Return process rank for the communicator.
int	size (MPI_Comm comm)
void	check_error (MPI_Comm comm, int code)
	Check MPI error code. If the error code is not equal to MPI_SUCCESS, then std::abort is called.
constexpr int	index_owner (int size, std::size_t index, std::size_t N)
	Return which rank owns index in global range [0, N - 1] (inverse of MPI::local_range).
std::vector< int >	compute_graph_edges_pcx (MPI_Comm comm, std::span< const int > edges)
	Determine incoming graph edges using the PCX consensus algorithm.
std::vector< int >	compute_graph_edges_nbx (MPI_Comm comm, std::span< const int > edges, int tagtag=static_cast< int >(tag::consensus_nbx))
	Determine incoming graph edges using the NBX consensus algorithm.
std::pair< std::vector< int >, std::vector< int > >	compute_graph_edges_nbx (MPI_Comm comm, std::span< const int > edges0, int tag0, std::span< const int > edges1, int tag1)
	Determine incoming graph edges for two independent edge sets using a single, overlapped NBX consensus round.
template<std::ranges::contiguous_range U>
std::pair< std::vector< std::int32_t >, std::vector< std::ranges::range_value_t< U > > >	distribute_to_postoffice (MPI_Comm comm, const U &x, std::array< std::int64_t, 2 > shape, std::int64_t rank_offset)
	Send row data to its 'post office' rank.
template<std::ranges::contiguous_range U>
std::vector< std::ranges::range_value_t< U > >	distribute_from_postoffice (MPI_Comm comm, std::span< const std::int64_t > indices, const U &x, std::array< std::int64_t, 2 > shape, std::int64_t rank_offset)
	Fetch rows of a distributed row-major array via their post office ranks.
template<std::ranges::contiguous_range U>
std::vector< std::ranges::range_value_t< U > >	distribute_data (MPI_Comm comm0, std::span< const std::int64_t > indices, MPI_Comm comm1, const U &x, int shape1)
	Distribute rows of a row-major array to the ranks that require them, via the post office pattern.

Variables
template<typename T>
MPI_Datatype	mpi_t = mpi_datatype<T>()
	Retrieves the MPI data type associated to the provided type.

Detailed Description

MPI support functionality.

Function Documentation

◆ check_error()

void check_error	(	MPI_Comm	comm,
		int	code )

Check MPI error code. If the error code is not equal to MPI_SUCCESS, then std::abort is called.

Parameters

[in]	comm	MPI communicator.
[in]	code	Error code returned by an MPI function call.

◆ compute_graph_edges_nbx() [1/2]

std::vector< int > compute_graph_edges_nbx	(	MPI_Comm	comm,
		std::span< const int >	edges,
		int	tag = static_cast<int>(tag::consensus_nbx) )

Determine incoming graph edges using the NBX consensus algorithm.

Given a list of outgoing edges (destination ranks) from this rank, this function returns the incoming edges (source ranks) to this rank.

Note: This function is for sparse communication patterns, i.e. where the number of ranks that communicate with each other is relatively small. It is scalable, i.e. no arrays the size of the communicator are constructed and the communication pattern is sparse. It implements the NBX algorithm presented in https://dx.doi.org/10.1145/1837853.1693476.; The order of the returned ranks is not deterministic.; Collective.

Parameters

[in]	comm	MPI communicator
[in]	edges	Edges (ranks) from this rank (the caller).

Precondition: edges must not contain duplicate ranks.

Parameters

[in] tag Tag used in non-blocking MPI calls. A tag can be required when this function is called a second time on some ranks before a previous call has completed on all other ranks.

Returns: Ranks that have defined edges from them to this rank.

Note: An alternative to passing a tag is to ensure that there is an implicit or explicit barrier before and after the call to this function.

◆ compute_graph_edges_nbx() [2/2]

std::pair< std::vector< int >, std::vector< int > > compute_graph_edges_nbx	(	MPI_Comm	comm,
		std::span< const int >	edges0,
		int	tag0,
		std::span< const int >	edges1,
		int	tag1 )

Determine incoming graph edges for two independent edge sets using a single, overlapped NBX consensus round.

Equivalent to calling MPI::compute_graph_edges_nbx independently for edges0 and edges1, but runs both consensus rounds concurrently behind a single shared barrier rather than one after the other, halving the number of global barrier round-trips when the two edge sets can be computed without waiting on one another.

Note: tag0 and tag1 must differ from one another, and from any tag used by another consensus round that may be in flight concurrently on comm.; Collective.

Parameters

[in]	comm	MPI communicator
[in]	edges0	Edges (ranks) from this rank (the caller) for the first edge set.
[in]	tag0	Tag used for edges0's messages.
[in]	edges1	Edges (ranks) from this rank (the caller) for the second edge set.
[in]	tag1	Tag used for edges1's messages.

Returns: Ranks with defined edges to this rank, for (0) edges0 and (1) edges1.

◆ compute_graph_edges_pcx()

std::vector< int > compute_graph_edges_pcx	(	MPI_Comm	comm,
		std::span< const int >	edges )

Determine incoming graph edges using the PCX consensus algorithm.

Given a list of outgoing edges (destination ranks) from this rank, this function returns the incoming edges (source ranks) to this rank.

Note: This function is for sparse communication patterns, i.e. where the number of ranks that communicate with each other is relatively small. It is not scalable as arrays the size of the communicator are allocated. It implements the PCX algorithm described in https://dx.doi.org/10.1145/1837853.1693476.; For sparse graphs, this function has \(O(p)\) cost, where \(p\)is the number of MPI ranks. It is suitable for modest MPI rank counts.; The order of the returned ranks is not deterministic.; Collective

Parameters

[in]	comm	MPI communicator
[in]	edges	Edges (ranks) from this rank (the caller).

Precondition: edges must not contain duplicate ranks (a duplicate can leave a message unconsumed on the shared PCX tag, corrupting a later call).

Returns: Ranks that have defined edges from them to this rank.

◆ distribute_data()

template<std::ranges::contiguous_range U>

std::vector< std::ranges::range_value_t< U > > distribute_data	(	MPI_Comm	comm0,
		std::span< const std::int64_t >	indices,
		MPI_Comm	comm1,
		const U &	x,
		int	shape1 )

Distribute rows of a row-major array to the ranks that require them, via the post office pattern.

Scalable provided each rank exchanges with only a modest number of others.

Parameters

[in]	comm0	Communicator over which indices are resolved and the result is returned.
[in]	indices	Global row indices required by the calling rank.
[in]	comm1	Communicator across which x is distributed – typically comm0 itself or a sub-communicator of it, and MPI_COMM_NULL on ranks where x is empty.
[in]	x	Local block of rows to distribute (row-major); local row i has global index given by an exclusive scan of local row counts over comm1.
[in]	shape1	Number of columns of x.

Returns: The row for each entry of indices, in the same order (row-major storage).

Precondition: shape1 > 0.

◆ distribute_from_postoffice()

template<std::ranges::contiguous_range U>

std::vector< std::ranges::range_value_t< U > > distribute_from_postoffice	(	MPI_Comm	comm,
		std::span< const std::int64_t >	indices,
		const U &	x,
		std::array< std::int64_t, 2 >	shape,
		std::int64_t	rank_offset )

Fetch rows of a distributed row-major array via their post office ranks.

x is a contiguous local block of a larger row-major array distributed over comm, with local row i at global index rank_offset + i. For each global row index in indices (which may contain repeats), returns that row – read directly from x if already local, otherwise requested from its post office rank via MPI neighbourhood collectives. Scalable provided each rank exchanges with only a modest number of others.

Parameters

[in]	comm	MPI communicator.
[in]	indices	Global row indices required by the caller.
[in]	x	Local block of the array to distribute (row-major).
[in]	shape	Global shape of the array, {num_rows, num_cols}.
[in]	rank_offset	Global row index of local row 0 in x (usually an exclusive scan of row counts over the communicator x is distributed across, which may differ from comm).

Returns: The row for each entry of indices, in the same order (row-major storage).

Precondition: shape[1] > 0.

◆ distribute_to_postoffice()

template<std::ranges::contiguous_range U>

std::pair< std::vector< std::int32_t >, std::vector< std::ranges::range_value_t< U > > > distribute_to_postoffice	(	MPI_Comm	comm,
		const U &	x,
		std::array< std::int64_t, 2 >	shape,
		std::int64_t	rank_offset )

Send row data to its 'post office' rank.

x is a contiguous local block of a larger row-major array distributed over comm, with local row i at global index rank_offset + i. Each row is sent to its post office rank (dolfinx::MPI::index_owner applied to the row's global index and shape[0]), except rows for which the caller is itself the post office, which are left in place.

Parameters

[in]	comm	MPI communicator.
[in]	x	Local block of the array to distribute (row-major).
[in]	shape	Global shape of the array, {num_rows, num_cols}.
[in]	rank_offset	Global row index of local row 0 in x, usually computed with MPI_Exscan.

Returns: (0) position of each received row within the caller's post-office partition of [0, shape[0]), and (1) the received row data (row-major). Rows for which the caller is itself the post office are not included.

◆ index_owner()

int index_owner	(	int	size,
		std::size_t	index,
		std::size_t	N )

constexpr

Return which rank owns index in global range [0, N - 1] (inverse of MPI::local_range).

Parameters

[in]	size	Number of MPI ranks.
[in]	index	The index to determine the owning rank of.
[in]	N	Total number of indices.

Returns: Rank of the owning process.

◆ size()

int size ( MPI_Comm comm )

Return size of the group (number of processes) associated with the communicator.

Variable Documentation

◆ mpi_t

template<typename T>

MPI_Datatype mpi_t = mpi_datatype<T>()

Retrieves the MPI data type associated to the provided type.

Template Parameters

T	cpp type to map

Classes

Enumerations

Functions

Variables

Detailed Description

Function Documentation

◆ check_error()

◆ compute_graph_edges_nbx() [1/2]

◆ compute_graph_edges_nbx() [2/2]

◆ compute_graph_edges_pcx()

◆ distribute_data()

◆ distribute_from_postoffice()

◆ distribute_to_postoffice()

◆ index_owner()

◆ size()

Variable Documentation

◆ mpi_t