A Scatterer supports the scattering and gathering of distributed data that is associated with a common::IndexMap, using MPI. More...

#include <Scatterer.h>

Public Types
using	container_type = Container
	Container type used to store local and remote indices.

Public Member Functions
	Scatterer (const IndexMap &map, int bs)
	Create a scatterer for data with a layout described by an IndexMap and a block size.
	Scatterer (const Scatterer &scatterer)=default
	Copy constructor.
template<class U>
	Scatterer (const Scatterer< U > &s)
	Cast-copy constructor.
template<typename T>
void	scatter_fwd_begin (const T send_buffer, T recv_buffer, MPI_Request &request) const
	Start a non-blocking send of owned data to ranks that ghost the data using MPI neighbourhood collective communication (recommended).
template<typename T>
void	scatter_fwd_begin (const T send_buffer, T recv_buffer, std::span< MPI_Request > requests) const
	Start a non-blocking send of owned data to ranks that ghost the data using point-to-point MPI communication.
template<typename T>
void	scatter_rev_begin (const T send_buffer, T recv_buffer, MPI_Request &request) const
	Start a non-blocking send of ghost data to ranks that own the data using MPI neighbourhood collective communication (recommended).
template<typename T>
void	scatter_rev_begin (const T send_buffer, T recv_buffer, std::span< MPI_Request > requests) const
	Start a non-blocking send of ghost data to ranks that own the data using point-to-point MPI communication.
void	scatter_end (std::span< MPI_Request > requests) const
	Complete non-blocking MPI point-to-point sends.
void	scatter_end (MPI_Request &request) const
	Complete a non-blocking MPI neighbourhood collective send.
const container_type &	local_indices () const noexcept
	Array of indices for packing/unpacking owned data to/from a send/receive buffer.
const container_type &	remote_indices () const noexcept
	Array of indices for packing/unpacking ghost data to/from a send/receive buffer.
std::size_t	num_p2p_requests ()
	Number of required MPI_Requests for point-to-point communication.

Detailed Description

template<class Container = std::vector<std::int32_t>>
class dolfinx::common::Scatterer< Container >

A Scatterer supports the scattering and gathering of distributed data that is associated with a common::IndexMap, using MPI.

Scatter and gather operations can use:

MPI neighbourhood collectives (recommended), or
Non-blocking point-to-point communication modes.

The implementation is designed for sparse communication patterns, as is typical of patterns based on an IndexMap.

A Scatterer is stateless, i.e. it provides the required information and static data for a given parallel communication pattern but does not provide any communication caches or track the status of MPI requests. Callers of the a Scatterer's members are responsible for managing buffer and MPI request handles.

Template Parameters

Container Container type for storing the 'local' and 'remote' indices. On CPUs this is normally std::vector<std::int32_t>. For GPUs the container should store the indices on the device, e.g. using thrust::device_vector<std::int32_t>.

Constructor & Destructor Documentation

◆ Scatterer() [1/2]

template<class Container = std::vector<std::int32_t>>

Scatterer	(	const IndexMap &	map,
		int	bs )

inline

Create a scatterer for data with a layout described by an IndexMap and a block size.

Parameters

[in]	map	Index map that describes the parallel layout of data.
[in]	bs	Number of values associated with each map index (the block size).

◆ Scatterer() [2/2]

template<class Container = std::vector<std::int32_t>>

template<class U>

Scatterer ( const Scatterer< U > & s )

inline

Cast-copy constructor.

Create a copy of a Scatterer, were the copy uses a different storage container for indices that are used in MPI communication. Example usage includes creating from a CPU-suitable Scatterer a GPU-suitable Scatterer that can be used with GPU-aware MPI to move data between devices. This would be typical when copying a la::Vector or la::MatrixCSR to/from a GPU. When copying a vector or matrix to/from a GPU, the underlying Scatter that manages parallel communication will usually be copied too with a different storage container.

Parameters

s	Scatterer to copy

Member Function Documentation

◆ local_indices()

template<class Container = std::vector<std::int32_t>>

const container_type & local_indices ( ) const

inlinenoexcept

Array of indices for packing/unpacking owned data to/from a send/receive buffer.

For a forward scatter, the indices are used to copy required entries in the owned part of the data array into the appropriate position in a send buffer. For a reverse scatter, indices are used for assigning (accumulating) the receive buffer values into correct position in the owned part of the data array.

For a forward scatter, if x is the owned part of an array and send_buffer is the send buffer, send_buffer is packed such that:

auto& idx = scatterer.local_indices()
std::vector<T> send_buffer(idx.size())
for (std::size_t i = 0; i < idx.size(); ++i)
    send_buffer[i] = x[idx[i]];

For a reverse scatter, if recv_buffer is the received buffer, then x is updated by

auto& idx = scatterer.local_indices()
std::vector<T> recv_buffer(idx.size())
for (std::size_t i = 0; i < idx.size(); ++i)
    x[idx[i]] = op(recv_buffer[i], x[idx[i]]);

where op is a binary operation, e.g. x[idx[i]] = buffer[i] or x[idx[i]] += buffer[i].

Returns: Indices container.

◆ num_p2p_requests()

template<class Container = std::vector<std::int32_t>>

std::size_t num_p2p_requests ( )

inline

Number of required MPI_Requests for point-to-point communication.

Returns: Numer of required MPI request handles.

◆ remote_indices()

template<class Container = std::vector<std::int32_t>>

const container_type & remote_indices ( ) const

inlinenoexcept

Array of indices for packing/unpacking ghost data to/from a send/receive buffer.

For a forward scatter, the indices are to copy required entries in the owned array into the appropriate position in a send buffer. For a reverse scatter, indices are used for assigning (accumulating) the receive buffer values to correct position in the owned array.

For a forward scatter, if xg is the ghost part of the data array and recv_buffer is the receive buffer, xg is updated that

auto& idx = scatterer.remote_indices()
std::vector<T> recv_buffer(idx.size())
for (std::size_t i = 0; i < idx.size(); ++i)
    xg[idx[i]] = recv_buffer[i];

For a reverse scatter, if send_buffer is the send buffer, then send_buffer is packaged such that:

auto& idx = scatterer.local_indices()
std::vector<T> send_buffer(idx.size())
for (std::size_t i = 0; i < idx.size(); ++i)
    send_buffer[i] = xg[idx[i]];

Returns: Indices container.

◆ scatter_end() [1/2]

template<class Container = std::vector<std::int32_t>>

void scatter_end ( MPI_Request & request ) const

inline

Complete a non-blocking MPI neighbourhood collective send.

This function completes the communication started by scatter_fwd_begin or scatter_rev_begin.

Parameters

[in] request MPI request handle for tracking the status of the send.

◆ scatter_end() [2/2]

template<class Container = std::vector<std::int32_t>>

void scatter_end ( std::span< MPI_Request > requests ) const

inline

Complete non-blocking MPI point-to-point sends.

This function completes the communication started by scatter_fwd_begin or scatter_rev_begin.

Parameters

[in] requests MPI request handles for tracking the status of sends.

◆ scatter_fwd_begin() [1/2]

template<class Container = std::vector<std::int32_t>>

template<typename T>

void scatter_fwd_begin	(	const T *	send_buffer,
		T *	recv_buffer,
		MPI_Request &	request ) const

inline

Start a non-blocking send of owned data to ranks that ghost the data using MPI neighbourhood collective communication (recommended).

The communication is completed by calling Scatterer::scatter_end. See local_indices for instructions on packing send_buffer and remote_indices for instructions on unpacking recv_buffer.

Note: The send and receive buffers must not to be changed or accessed until after a call to Scatterer::scatter_end.; The pointers send_buffer and recv_buffer must be pointers to the data on the target device. E.g., if the send and receive buffers are allocated on a GPU, the send_buffer and recv_buffer should be device pointers.

Parameters

[in]	send_buffer	Packed local data associated with each owned local index to be sent to processes where the data is ghosted. See Scatterer::local_indices for the order of the buffer and how to pack.
[in,out]	recv_buffer	Buffer for storing received data. See Scatterer::remote_indices for the order of the buffer and how to unpack.
[in]	request	MPI request handle for tracking the status of the non-blocking communication. The same request handle should be passed to Scatterer::scatter_end to complete the communication.

◆ scatter_fwd_begin() [2/2]

template<class Container = std::vector<std::int32_t>>

template<typename T>

void scatter_fwd_begin	(	const T *	send_buffer,
		T *	recv_buffer,
		std::span< MPI_Request >	requests ) const

inline

Start a non-blocking send of owned data to ranks that ghost the data using point-to-point MPI communication.

See scatter_fwd_begin for a detailed explanation of usage, including on the send and receive buffer packing and unpacking

Note: Use of the neighbourhood version of scatter_fwd_begin is recommended over this version.

Parameters

[in]	send_buffer	Send buffer.
[in,out]	recv_buffer	Receive buffer.
[in]	requests	List of MPI request handles. The length of the list must be num_p2p_requests()

◆ scatter_rev_begin() [1/2]

template<class Container = std::vector<std::int32_t>>

template<typename T>

void scatter_rev_begin	(	const T *	send_buffer,
		T *	recv_buffer,
		MPI_Request &	request ) const

inline

Start a non-blocking send of ghost data to ranks that own the data using MPI neighbourhood collective communication (recommended).

The communication is completed by calling Scatterer::scatter_end. See remote_indices for instructions on packing send_buffer and local_indices for instructions on unpacking recv_buffer.

Note: The send and receive buffers must not to be changed or accessed until after a call to Scatterer::scatter_end.; The pointers send_buffer and recv_buffer must be pointers to the data on the target device. E.g., if the send and receive buffers are allocated on a GPU, the send_buffer and recv_buffer should be device pointers.

Parameters

[in]	send_buffer	Data associated with each ghost index. This data is sent to process that owns the index. See Scatterer::remote_indices for the order of the buffer and how to pack.
[in,out]	recv_buffer	Buffer for storing received data. See Scatterer::local_indices for the order of the buffer and how to unpack.
[in]	request	MPI request handle for tracking the status of the non-blocking communication. The same request handle should be passed to Scatterer::scatter_end to complete the communication.

◆ scatter_rev_begin() [2/2]

template<class Container = std::vector<std::int32_t>>

template<typename T>

void scatter_rev_begin	(	const T *	send_buffer,
		T *	recv_buffer,
		std::span< MPI_Request >	requests ) const

inline

Start a non-blocking send of ghost data to ranks that own the data using point-to-point MPI communication.

See scatter_rev_begin for a detailed explanation of usage, including on the send and receive buffer packing and unpacking

Note: Use of the neighbourhood version of scatter_rev_begin is recommended over this version

Parameters

[in]	send_buffer	Send buffer.
[in,out]	recv_buffer	Receive buffer.
[in]	requests	List of MPI request handles. The length of the list must be num_p2p_requests()

The documentation for this class was generated from the following file:

/__w/dolfinx/dolfinx/cpp/dolfinx/common/Scatterer.h

Public Types

Public Member Functions

Detailed Description

Constructor & Destructor Documentation

◆ Scatterer() [1/2]

◆ Scatterer() [2/2]

Member Function Documentation

◆ local_indices()

◆ num_p2p_requests()

◆ remote_indices()

◆ scatter_end() [1/2]

◆ scatter_end() [2/2]

◆ scatter_fwd_begin() [1/2]

◆ scatter_fwd_begin() [2/2]

◆ scatter_rev_begin() [1/2]

◆ scatter_rev_begin() [2/2]