A Scatterer supports the MPI scattering and gathering of data that is associated with a common::IndexMap. More...

#include <Scatterer.h>

Public Member Functions
	Scatterer (const IndexMap &map, int bs)
	Create a scatterer. More...

template<typename T >
void	scatter_fwd_begin (const std::span< const T > &send_buffer, const std::span< T > &recv_buffer, MPI_Request &request) const
	Start a non-blocking send of owned data to ranks that ghost the data. More...

void	scatter_fwd_end (MPI_Request &request) const
	Complete a non-blocking send from the local owner to process ranks that have the index as a ghost. More...

template<typename T , typename Functor >
void	scatter_fwd_begin (const std::span< const T > &local_data, std::span< T > local_buffer, std::span< T > remote_buffer, Functor pack_fn, MPI_Request &request) const
	Scatter data associated with owned indices to ghosting ranks. More...

template<typename T , typename Functor >
void	scatter_fwd_end (const std::span< const T > &remote_buffer, std::span< T > remote_data, Functor unpack_fn, MPI_Request &request) const
	Complete a non-blocking send from the local owner to process ranks that have the index as a ghost, and unpack received buffer into remote data. More...

template<typename T >
void	scatter_fwd (const std::span< const T > &local_data, std::span< T > remote_data) const
	Scatter data associated with owned indices to ghosting ranks. More...

template<typename T >
void	scatter_rev_begin (const std::span< const T > &send_buffer, const std::span< T > &recv_buffer, MPI_Request &request) const
	Start a non-blocking send of ghost data to ranks that own the data. More...

void	scatter_rev_end (MPI_Request &request) const
	End the reverse scatter communication. More...

template<typename T , typename Functor >
void	scatter_rev_begin (const std::span< const T > &remote_data, std::span< T > remote_buffer, std::span< T > local_buffer, Functor pack_fn, MPI_Request &request) const
	Scatter data associated with ghost indices to owning ranks. More...

template<typename T , typename Functor , typename BinaryOp >
void	scatter_rev_end (const std::span< const T > &local_buffer, std::span< T > local_data, Functor unpack_fn, BinaryOp op, MPI_Request &request)
	End the reverse scatter communication, and unpack the received local buffer into local data. More...

template<typename T , typename BinaryOp >
void	scatter_rev (std::span< T > local_data, const std::span< const T > &remote_data, BinaryOp op)
	Scatter data associated with ghost indices to ranks that own the indices.

std::int32_t	local_buffer_size () const noexcept
	Size of buffer for local data (owned and shared) used in forward and reverse communication. More...

std::int32_t	remote_buffer_size () const noexcept
	Buffer size for remote data (ghosts) used in forward and reverse communication. More...

const std::vector< std::int32_t > &	local_indices () const noexcept
	Return a vector of local indices (owned) used to pack/unpack local data. These indices are grouped by neighbor process (process for which an index is a ghost).

const std::vector< std::int32_t > &	remote_indices () const noexcept
	Return a vector of remote indices (ghosts) used to pack/unpack ghost data. These indices are grouped by neighbor process (ghost owners).

int	bs () const noexcept
	The number values (block size) to send per index in the common::IndexMap use to create the scatterer. More...

Detailed Description

A Scatterer supports the MPI scattering and gathering of data that is associated with a common::IndexMap.

Scatter and gather operations uses MPI neighbourhood collectives. The implementation is designed is for sparse communication patterns, as it typical of patterns based on and IndexMap.

Constructor & Destructor Documentation

◆ Scatterer()

Scatterer	(	const IndexMap &	map,
		int	bs
	)

Create a scatterer.

Parameters

[in]	map	The index map that describes the parallel layout of data
[in]	bs	The block size of data associated with each index in `map` that will be scattered/gathered

Member Function Documentation

◆ bs()

int bs ( ) const

noexcept

The number values (block size) to send per index in the common::IndexMap use to create the scatterer.

Returns: The block size

◆ local_buffer_size()

std::int32_t local_buffer_size ( ) const

noexcept

Size of buffer for local data (owned and shared) used in forward and reverse communication.

Returns: The required buffer size

◆ remote_buffer_size()

std::int32_t remote_buffer_size ( ) const

noexcept

Buffer size for remote data (ghosts) used in forward and reverse communication.

Returns: The required buffer size

◆ scatter_fwd()

void scatter_fwd	(	const std::span< const T > &	local_data,
		std::span< T >	remote_data
	)		const

inline

Scatter data associated with owned indices to ghosting ranks.

Parameters

[in]	local_data	All data associated with owned indices. Size is `size_local()` from the IndexMap used to create the scatterer, multiplied by the block size. The data for each index is blocked
[out]	remote_data	Received data associated with the ghost indices. The order follows the order of the ghost indices in the IndexMap used to create the scatterer. The size equal to the number of ghosts in the index map multiplied by the block size. The data for each index is blocked.

◆ scatter_fwd_begin() [1/2]

void scatter_fwd_begin	(	const std::span< const T > &	local_data,
		std::span< T >	local_buffer,
		std::span< T >	remote_buffer,
		Functor	pack_fn,
		MPI_Request &	request
	)		const

inline

Scatter data associated with owned indices to ghosting ranks.

Note: This function is intended for advanced usage, and in particular when using CUDA/device-aware MPI.

Parameters

[in]	local_data	All data associated with owned indices. Size is `size_local()` from the IndexMap used to create the scatterer, multiplied by the block size. The data for each index is blocked.
[in]	local_buffer	Working buffer. The required size is given by Scatterer::local_buffer_size.
[out]	remote_buffer	Working buffer. The required size is given by Scatterer::remote_buffer_size.
[in]	pack_fn	Function to pack data from `local_data` into the send buffer. It is passed as an argument to support CUDA/device-aware MPI.
[in]	request	The MPI request handle for tracking the status of the send

◆ scatter_fwd_begin() [2/2]

void scatter_fwd_begin	(	const std::span< const T > &	send_buffer,
		const std::span< T > &	recv_buffer,
		MPI_Request &	request
	)		const

inline

Start a non-blocking send of owned data to ranks that ghost the data.

The communication is completed by calling Scatterer::scatter_fwd_end. The send and receive buffer should not be changed until after Scatterer::scatter_fwd_end has been called.

Parameters

[in]	send_buffer	Local data associated with each owned local index to be sent to process where the data is ghosted. It must not be changed until after a call to Scatterer::scatter_fwd_end. The order of data in the buffer is given by Scatterer::local_indices.
	recv_buffer	A buffer used for the received data. The position of ghost entries in the buffer is given by Scatterer::remote_indices. The buffer must not be accessed or changed until after a call to Scatterer::scatter_fwd_end.
	request	The MPI request handle for tracking the status of the non-blocking communication

◆ scatter_fwd_end() [1/2]

void scatter_fwd_end	(	const std::span< const T > &	remote_buffer,
		std::span< T >	remote_data,
		Functor	unpack_fn,
		MPI_Request &	request
	)		const

inline

Complete a non-blocking send from the local owner to process ranks that have the index as a ghost, and unpack received buffer into remote data.

This function completes the communication started by Scatterer::scatter_fwd_begin.

Parameters

[in]	remote_buffer	Working buffer, same used in Scatterer::scatter_fwd_begin.
[out]	remote_data	Received data associated with the ghost indices. The order follows the order of the ghost indices in the IndexMap used to create the scatterer. The size equal to the number of ghosts in the index map multiplied by the block size. The data for each index is blocked.
[in]	unpack_fn	Function to unpack the received buffer into `remote_data`. It is passed as an argument to support CUDA/device-aware MPI.
[in]	request	The MPI request handle for tracking the status of the send

◆ scatter_fwd_end() [2/2]

void scatter_fwd_end ( MPI_Request & request ) const

Complete a non-blocking send from the local owner to process ranks that have the index as a ghost.

This function completes the communication started by Scatterer::scatter_fwd_begin.

Parameters

[in] request The MPI request handle for tracking the status of the send

◆ scatter_rev_begin() [1/2]

void scatter_rev_begin	(	const std::span< const T > &	remote_data,
		std::span< T >	remote_buffer,
		std::span< T >	local_buffer,
		Functor	pack_fn,
		MPI_Request &	request
	)		const

inline

Scatter data associated with ghost indices to owning ranks.

Note: This function is intended for advanced usage, and in particular when using CUDA/device-aware MPI.

Template Parameters

T	The data type to send
BinaryOp	The reduction to perform when reducing data received from ghosting ranks to the value associated with the index on the owner
Functor1	The pack function
Functor2	The unpack function

Parameters

[in]	remote_data	Received data associated with the ghost indices. The order follows the order of the ghost indices in the IndexMap used to create the scatterer. The size equal to the number of ghosts in the index map multiplied by the block size. The data for each index is blocked.
[out]	local_buffer	Working buffer. The requires size is given by Scatterer::local_buffer_size.
[out]	remote_buffer	Working buffer. The requires size is given by Scatterer::remote_buffer_size.
[in]	pack_fn	Function to pack data from `local_data` into the send buffer. It is passed as an argument to support CUDA/device-aware MPI.
	request	The MPI request handle for tracking the status of the non-blocking communication

◆ scatter_rev_begin() [2/2]

void scatter_rev_begin	(	const std::span< const T > &	send_buffer,
		const std::span< T > &	recv_buffer,
		MPI_Request &	request
	)		const

inline

Start a non-blocking send of ghost data to ranks that own the data.

The communication is completed by calling Scatterer::scatter_rev_end. The send and receive buffers should not be changed until after Scatterer::scatter_rev_end has been called.

Parameters

[in]	send_buffer	Data associated with each ghost index. This data is sent to process that owns the index. It must not be changed until after a call to Scatterer::scatter_ref_end.
	recv_buffer	Buffer used for the received data. The position of owned indices in the buffer is given by Scatterer::local_indices. Scatterer::local_displacements()[i] is the location of the first entry in `recv_buffer` received from neighbourhood rank i. The number of entries received from neighbourhood rank i is Scatterer::local_displacements()[i + 1] - Scatterer::local_displacements()[i]. `recv_buffer[j]` is the data associated with the index Scatterer::local_indices()[j] in the index map.

The buffer must not be accessed or changed until after a call to Scatterer::scatter_fwd_end.

Parameters

request The MPI request handle for tracking the status of the non-blocking communication

◆ scatter_rev_end() [1/2]

void scatter_rev_end	(	const std::span< const T > &	local_buffer,
		std::span< T >	local_data,
		Functor	unpack_fn,
		BinaryOp	op,
		MPI_Request &	request
	)

inline

End the reverse scatter communication, and unpack the received local buffer into local data.

This function must be called after Scatterer::scatter_rev_begin. The buffers passed to Scatterer::scatter_rev_begin must not be modified until after the function has been called.

Parameters

[in]	local_buffer	Working buffer. Same buffer should be used in Scatterer::scatter_rev_begin.
[out]	local_data	All data associated with owned indices. Size is `size_local()` from the IndexMap used to create the scatterer, multiplied by the block size. The data for each index is blocked.
[in]	unpack_fn	Function to unpack the receive buffer into `local_data`. It is passed as an argument to support CUDA/device-aware MPI.
[in]	op	The reduction operation when accumulating received values. To add the received values use `std::plus<T>()`.
[in]	request	The handle used when calling Scatterer::scatter_rev_begin

◆ scatter_rev_end() [2/2]

void scatter_rev_end ( MPI_Request & request ) const

End the reverse scatter communication.

This function must be called after Scatterer::scatter_rev_begin. The buffers passed to Scatterer::scatter_rev_begin must not be modified until after the function has been called.

Parameters

[in] request The handle used when calling Scatterer::scatter_rev_begin

The documentation for this class was generated from the following files:

/__w/dolfinx/dolfinx/cpp/dolfinx/common/Scatterer.h
/__w/dolfinx/dolfinx/cpp/dolfinx/common/Scatterer.cpp

Public Member Functions

Detailed Description

Constructor & Destructor Documentation

◆ Scatterer()

Member Function Documentation

◆ bs()

◆ local_buffer_size()

◆ remote_buffer_size()

◆ scatter_fwd()

◆ scatter_fwd_begin() [1/2]

◆ scatter_fwd_begin() [2/2]

◆ scatter_fwd_end() [1/2]

◆ scatter_fwd_end() [2/2]

◆ scatter_rev_begin() [1/2]

◆ scatter_rev_begin() [2/2]

◆ scatter_rev_end() [1/2]

◆ scatter_rev_end() [2/2]