MPI support functionality. More...

Classes
class	Comm
	A duplicate MPI communicator and manage lifetime of the communicator. More...

struct	dependent_false

Enumerations
enum class	tag : int { consensus_pcx , consensus_pex }
	MPI communication tags.

Functions
int	rank (MPI_Comm comm)
	Return process rank for the communicator.

int	size (MPI_Comm comm)

void	check_error (MPI_Comm comm, int code)
	Check MPI error code. If the error code is not equal to MPI_SUCCESS, then std::abort is called.

constexpr std::array< std::int64_t, 2 >	local_range (int rank, std::int64_t N, int size)
	Return local range for the calling process, partitioning the global [0, N - 1] range across all ranks into partitions of almost equal size.

constexpr int	index_owner (int size, std::size_t index, std::size_t N)
	Return which rank owns index in global range [0, N - 1] (inverse of MPI::local_range).

std::vector< int >	compute_graph_edges_pcx (MPI_Comm comm, std::span< const int > edges)
	Determine incoming graph edges using the PCX consensus algorithm.

std::vector< int >	compute_graph_edges_nbx (MPI_Comm comm, std::span< const int > edges)
	Determine incoming graph edges using the NBX consensus algorithm.

template<typename U >
std::pair< std::vector< std::int32_t >, std::vector< typename std::remove_reference_t< typename U::value_type > > >	distribute_to_postoffice (MPI_Comm comm, const U &x, std::array< std::int64_t, 2 > shape, std::int64_t rank_offset)
	Distribute row data to 'post office' ranks.

template<typename U >
std::vector< typename std::remove_reference_t< typename U::value_type > >	distribute_from_postoffice (MPI_Comm comm, std::span< const std::int64_t > indices, const U &x, std::array< std::int64_t, 2 > shape, std::int64_t rank_offset)
	Distribute rows of a rectangular data array from post office ranks to ranks where they are required.

template<typename U >
std::vector< typename std::remove_reference_t< typename U::value_type > >	distribute_data (MPI_Comm comm0, std::span< const std::int64_t > indices, MPI_Comm comm1, const U &x, int shape1)
	Distribute rows of a rectangular data array to ranks where they are required (scalable version).

template<typename T >
constexpr MPI_Datatype	mpi_type ()
	MPI Type.

Detailed Description

MPI support functionality.

Function Documentation

◆ check_error()

void check_error	(	MPI_Comm	comm,
		int	code )

Check MPI error code. If the error code is not equal to MPI_SUCCESS, then std::abort is called.

Parameters

[in]	comm	MPI communicator
[in]	code	Error code returned by an MPI function call

◆ compute_graph_edges_nbx()

std::vector< int > compute_graph_edges_nbx	(	MPI_Comm	comm,
		std::span< const int >	edges )

Determine incoming graph edges using the NBX consensus algorithm.

Given a list of outgoing edges (destination ranks) from this rank, this function returns the incoming edges (source ranks) to this rank.

Note: This function is for sparse communication patterns, i.e. where the number of ranks that communicate with each other is relatively small. It is scalable, i.e. no arrays the size of the communicator are constructed and the communication pattern is sparse. It implements the NBX algorithm presented in https://dx.doi.org/10.1145/1837853.1693476.; For sparse graphs, this function has \(O(\log p)\) cost, where \(p\)is the number of MPI ranks. It is suitable for modest MPI rank counts.; The order of the returned ranks is not deterministic.; Collective.

Parameters

[in]	comm	MPI communicator
[in]	edges	Edges (ranks) from this rank (the caller).

Returns: Ranks that have defined edges from them to this rank.

◆ compute_graph_edges_pcx()

std::vector< int > compute_graph_edges_pcx	(	MPI_Comm	comm,
		std::span< const int >	edges )

Determine incoming graph edges using the PCX consensus algorithm.

Given a list of outgoing edges (destination ranks) from this rank, this function returns the incoming edges (source ranks) to this rank.

Note: This function is for sparse communication patterns, i.e. where the number of ranks that communicate with each other is relatively small. It is not scalable as arrays the size of the communicator are allocated. It implements the PCX algorithm described in https://dx.doi.org/10.1145/1837853.1693476.; For sparse graphs, this function has \(O(p)\) cost, where \(p\)is the number of MPI ranks. It is suitable for modest MPI rank counts.; The order of the returned ranks is not deterministic.; Collective

Parameters

[in]	comm	MPI communicator
[in]	edges	Edges (ranks) from this rank (the caller).

Returns: Ranks that have defined edges from them to this rank.

◆ distribute_data()

template<typename U >

std::vector< typename std::remove_reference_t< typename U::value_type > > distribute_data	(	MPI_Comm	comm0,
		std::span< const std::int64_t >	indices,
		MPI_Comm	comm1,
		const U &	x,
		int	shape1 )

Distribute rows of a rectangular data array to ranks where they are required (scalable version).

This function determines local neighborhoods for communication, and then uses MPI neighbourhood collectives to exchange data. It is scalable if the neighborhoods are relatively small, i.e. each process communicated with a modest number of other processes.

Parameters

[in]	comm0	Communicator to distribute data across.
[in]	indices	Global indices of the data (row indices) required by the calling process.
[in]	comm1	Communicator across which x is distributed. Can be `MPI_COMM_NULL` on ranks where `x` is empty.
[in]	x	Data (2D array, row-major) on calling process to be distributed (by row). The global index for the `[0, ..., n)` local rows is assumed to be the local index plus the offset for this rank on `comm1`.
[in]	shape1	The number of columns of the data array `x`.

Returns: The data for each index in indices (row-major storage).

Precondition: shape1 > 0

◆ distribute_from_postoffice()

template<typename U >

std::vector< typename std::remove_reference_t< typename U::value_type > > distribute_from_postoffice	(	MPI_Comm	comm,
		std::span< const std::int64_t >	indices,
		const U &	x,
		std::array< std::int64_t, 2 >	shape,
		std::int64_t	rank_offset )

Distribute rows of a rectangular data array from post office ranks to ranks where they are required.

This function determines local neighborhoods for communication, and then using MPI neighbourhood collectives to exchange data. It is scalable if the neighborhoods are relatively small, i.e. each process communicated with a modest number of other processes.

Parameters

[in]	comm	The MPI communicator.
[in]	indices	Global indices of the data (row indices) required by calling process.
[in]	x	Data (2D array, row-major) on calling process which may be distributed (by row). The global index for the `[0, ..., n)` local rows is assumed to be the local index plus the offset for this rank.
[in]	shape	The global shape of `x`.
[in]	rank_offset	The rank offset such that global index of local row `i` in `x` is `rank_offset + i`. It is usually computed using `MPI_Exscan` on `comm1` from MPI::distribute_data.

Returns: The data for each index in indices (row-major storage).

Precondition: shape1 > 0.

◆ distribute_to_postoffice()

template<typename U >

std::pair< std::vector< std::int32_t >, std::vector< typename std::remove_reference_t< typename U::value_type > > > distribute_to_postoffice	(	MPI_Comm	comm,
		const U &	x,
		std::array< std::int64_t, 2 >	shape,
		std::int64_t	rank_offset )

Distribute row data to 'post office' ranks.

This function takes row-wise data that is distributed across processes. Data is not duplicated across ranks. The global index of a row is its local row position plus the offset for the calling process. The post office rank for a row is determined by applying dolfinx::MPI::index_owner to the global index, and the row is then sent to the post office rank. The function returns that row data for which the caller is the post office.

Parameters

[in]	comm	MPI communicator.
[in]	x	Data to distribute (2D, row-major layout).
[in]	shape	The global shape of `x`.
[in]	rank_offset	The rank offset such that global index of local row `i` in `x` is `rank_offset + i`. It is usually computed using `MPI_Exscan`.

Returns: (0) local indices of my post office data and (1) the data (row-major). It does not include rows that are in x, i.e. rows for which the calling process is the post office.

◆ index_owner()

int index_owner	(	int	size,
		std::size_t	index,
		std::size_t	N )

constexpr

Return which rank owns index in global range [0, N - 1] (inverse of MPI::local_range).

Parameters

[in]	size	Number of MPI ranks
[in]	index	The index to determine owning rank
[in]	N	Total number of indices

Returns: The rank of the owning process

◆ local_range()

std::array< std::int64_t, 2 > local_range	(	int	rank,
		std::int64_t	N,
		int	size )

constexpr

Return local range for the calling process, partitioning the global [0, N - 1] range across all ranks into partitions of almost equal size.

Parameters

[in]	rank	MPI rank of the caller
[in]	N	The value to partition
[in]	size	The number of MPI ranks across which to partition `N`

◆ size()

int size ( MPI_Comm comm )

Return size of the group (number of processes) associated with the communicator

Classes

Enumerations

Functions

Detailed Description

Function Documentation

◆ check_error()

◆ compute_graph_edges_nbx()

◆ compute_graph_edges_pcx()

◆ distribute_data()

◆ distribute_from_postoffice()

◆ distribute_to_postoffice()

◆ index_owner()

◆ local_range()

◆ size()