Common (dolfinx::common)#

namespace common#

Miscellaneous classes, functions and types.

Generic tools.

This namespace provides utility type functions for managing subsystems, convenience classes and library-wide typedefs.

Enums

enum class IndexMapOrder : bool#

Control ghost-index ordering in sub-index maps.

Values:

enumerator preserve#

Preserve the ordering of ghost indices.

enumerator any#

Allow arbitrary ghost-index ordering.

Functions

std::vector<std::int32_t> compute_owned_indices(std::span<const std::int32_t> indices, const IndexMap &map)#

Return selected indices owned by the calling rank.

Includes locally owned entries in indices and entries selected as ghosts on other ranks.

For example, on two ranks, suppose rank 0 owns global indices [0, 1] and has global index 2 as local ghost index 2, while rank 1 owns global indices [2, 3] and has global index 1 as local ghost index 2. If both ranks pass local index [2], the results are [1] on rank 0 and [0] on rank 1: each owns the global index selected as a ghost by the other rank.

Note

Collective

Parameters:

  • indices – [in] Sorted unique local indices (owned or ghost) in [0, map.size_local() + map.num_ghosts()).

  • map – [in] The index map.

Throws:

std::invalid_argument – If the indices precondition is violated in a Developer build.

Pre:

indices is sorted, unique, and in range. This condition is checked in Developer builds; callers must ensure it in Release builds.

Returns:

Local indices owned by the calling rank.

std::tuple<std::int64_t, std::vector<std::int32_t>, std::vector<std::vector<std::int64_t>>, std::vector<std::vector<int>>> stack_index_maps(const std::vector<std::pair<std::reference_wrapper<const IndexMap>, int>> &maps)#

Compute layout data for a concatenated index map.

Locally owned entries remain owned by the caller. Ghost entries are grouped by input map in maps.

Note

Collective. Maps with a block size are unrolled.

Parameters:

maps – [in] Non-empty pairs of index maps and positive block sizes. All maps must use the same communicator.

Pre:

All ranks supply corresponding maps in the same order.

Returns:

(0) Global offset on the calling rank, (1) local offsets for owned entries in each map, (2) global ghost indices for each map, and (3) their owner ranks.

std::vector<std::int32_t> num_sharing_ranks(const IndexMap &map, std::span<const std::int32_t> indices, int bs)#

Number of ranks that share each of the given local indices.

A rank shares an index if it owns it or ghosts it, so the count is one for an index that is not on a subdomain boundary, and the sum of the counts over all ranks that hold an index is the same on each of them.

Note

Collective. Arguments must be locally valid on every rank. Invalid input on only some ranks may deadlock.

Note

An overload taking an existing Scatterer is declared in Scatterer.h, and avoids building one here.

Parameters:

  • map – [in] Index map describing the parallel layout.

  • indices – [in] Local indices, which may include ghosts. An index refers to block i / bs of map.

  • bs – [in] Block size relating indices to the blocks of map.

Throws:

  • std::invalid_argument – If bs is less than one.

  • std::out_of_range – If an entry of indices is not a local index of map.

Returns:

Number of sharing ranks, one entry per entry of indices.

std::tuple<IndexMap, std::vector<std::int32_t>, bool> create_sub_index_map(const IndexMap &imap, std::span<const std::int32_t> indices, IndexMapOrder order = IndexMapOrder::any)#

Create an index map from a subset of an existing map.

Note

Collective

Note

(2) is rank-local and is not reduced: it can be true on some ranks and false on others. A caller that requires ownership to be preserved must reduce it (e.g. MPI_Allreduce with MPI_LOR) before acting on it, since throwing on only some ranks would leave the others in a subsequent collective.

Parameters:

  • imap – [in] Parent map to create a new sub-map from.

  • indices – [in] Local indices in imap (owned and ghost) to include in the new index map.

  • order – [in] Control the order in which ghost indices appear in the new map.

Throws:

std::invalid_argument – If the indices precondition is violated in a Developer build.

Pre:

indices contains unique local indices in range. This condition is checked in Developer builds; callers must ensure it in Release builds.

Returns:

(0) New index map, (1) corresponding local indices in imap, and (2) whether any index acquired a new owner in the submap. An index selected only by ghosting ranks, and not by its owner, is given a new owner in the submap; (2) reports whether this happened.

std::tuple<std::vector<int>, std::vector<int>, std::vector<std::int32_t>> compute_sharing_neighbourhood(const IndexMap &imap)#

Compute the symmetric neighbourhood of ranks that share an owned or ghost index with the caller.

A rank shares a local index with the caller if it also holds the index, as owner or ghost. For an owned index, the sharing ranks are the ranks that ghost it. For a ghost index, they are its owner and the other ranks that ghost it; the latter need not be in IndexMap::src or IndexMap::dest.

The in-neighbourhood IndexMap::src (owners of the caller’s ghosts) and the out-neighbourhood IndexMap::dest (ranks ghosting the caller’s owned indices) differ in general. Sharing is symmetric: rank a shares an index with rank b if and only if b shares it with a. The returned ranks can therefore be both the sources and the destinations of a neighbourhood communicator.

Example on four ranks, where | separates owned and ghost entries: 

rank 0: local [0, 1]    | [2]    -> global [0, 1]    | [2]    (owner 1)
rank 1: local [0, 1, 2] | [3]    -> global [2, 3, 4] | [6]    (owner 2)
rank 2: local [0, 1]    | [2]    -> global [5, 6]    | [2]    (owner 1)
rank 3: local [0]       | [1, 2] -> global [7]       | [1, 5] (owners 0, 2)
The return values are: 
        src     dest    unique_ranks  data          offsets
rank 0: [1]     [3]     [1, 2, 3]     [2, 0, 1]     [0, 0, 1, 3]
rank 1: [2]     [0, 2]  [0, 2]        [0, 1, 1]     [0, 2, 2, 2, 3]
rank 2: [1]     [1, 3]  [0, 1, 3]     [2, 1, 0, 1]  [0, 1, 2, 4]
rank 3: [0, 2]  []      [0, 2]        [0, 1]        [0, 0, 1, 2]
Ranks 0 and 2 share global index 2, which both ghost, although neither is in the other’s IndexMap::src or IndexMap::dest. Rank 3 is in IndexMap::dest of ranks 0 and 2, but its own IndexMap::dest is empty. On rank 0, local index 1 occupies entry [offsets[1], offsets[2]) = [0, 1) of data, whose value 2 is the position of rank 3 in unique_ranks.

Note

Collective.

Parameters:

imap – [in] Index map.

Returns:

(0) unique_ranks: sorted ranks, other than the caller, that share at least one index with the caller. (1) data and (2) offsets: adjacency list from each local index to the ranks sharing it, given as positions in (0). offsets has imap.size_local() + imap.num_ghosts() + 1 entries, and the ranks sharing local index i occupy [offsets[i], offsets[i + 1]) of data. data() of (0) is not null, as required by some MPI implementations, even if (0) is empty.

constexpr std::array<std::int64_t, 2> local_range(int index, std::int64_t N, int size)#

Partition a global range [0, N - 1] across callers into non-overlapping sub-partitions of almost equal size. Returns the local partition for the caller. The local partition range.

Partitions [0, N) into size non-overlapping partitions [n_(i0}, n_(i1)), where i is index and n_(i1) == n_((i+1)0).

Parameters:

  • index – [in] Index of the partition to compute.

  • N – [in] Global range to partition.

  • size – [in] Number of partitions into which to partition N.

template<class Container>
std::vector<std::int32_t> num_sharing_ranks(const IndexMap &map, const Scatterer<Container> &sc, std::span<const std::int32_t> indices, int bs)#

Number of ranks that share each of the given local indices, reusing an existing Scatterer.

Equivalent to the IndexMap overload, but avoids building a Scatterer, which duplicates MPI communicators.

Note

Collective. Arguments must be locally valid on every rank. Invalid input on only some ranks may deadlock.

Parameters:

  • map – [in] Index map describing the parallel layout.

  • sc – [in] Scatterer for map.

  • indices – [in] Local indices, which may include ghosts. An index refers to block i / bs of map.

  • bs – [in] Block size relating indices to the blocks of map.

Throws:

  • std::invalid_argument – If bs is less than one.

  • std::out_of_range – If an entry of indices is not a local index of map.

Returns:

Number of sharing ranks, one entry per entry of indices.

template<std::ranges::input_range U, std::ranges::input_range V>
std::pair<std::vector<typename U::value_type>, std::vector<typename V::value_type>> sort_unique(const U &indices, const V &values)#

Sort two arrays based on the values in array indices.

Any duplicate indices and the corresponding value are removed. In the case of duplicates, the entry with the smallest value is retained.

Parameters:

  • indices – [in] Array of indices.

  • values – [in] Array of values.

Returns:

Sorted (indices, values), with sorting based on indices.

template<class T>
std::size_t hash_local(const T &x)#

Compute a hash of a given object.

The hash is computed using Boost container hash (https://www.boost.org/doc/libs/release/libs/container_hash/).

Parameters:

x – [in] The object to compute a hash of.

Returns:

The hash values.

template<class T>
std::size_t hash_global(MPI_Comm comm, const T &x)#

Compute a hash for a distributed (MPI) object.

A hash is computed on each process for the local part of the object. Then, a hash of the std::vector containing each local hash key in rank order is returned.

Note

Collective

Parameters:

  • comm – [in] The communicator on which to compute the hash.

  • x – [in] The object to compute a hash of.

Returns:

The hash values.

class IndexMap#
#include <IndexMap.h>

Distribution of a global index range [0, N) across MPI ranks.

Each rank owns a contiguous global range. Local indices in [0, size_local()) address owned entries; remaining local indices address ghost entries.

Public Functions

IndexMap(MPI_Comm comm, std::int32_t local_size)#

Create a non-overlapping index map.

For example, if rank 0 has local_size = 2 and rank 1 has local_size = 3, the layout is: 

rank 0: local [0, 1]    -> global [0, 1]
rank 1: local [0, 1, 2] -> global [2, 3, 4]

Note

Collective

Parameters:

  • comm – [in] Communicator that the index map is distributed across.

  • local_size – [in] Number of owned entries. Must be non-negative.

Throws:

std::invalid_argument – If local_size is negative.

Pre:

Every rank in this collective call supplies a non-negative local_size. A violation on only some ranks may deadlock in Release builds.

Pre:

local_size is non-negative; this is always checked locally (no MPI communication).

IndexMap(MPI_Comm comm, std::int32_t local_size, std::span<const std::int64_t> ghosts, std::span<const int> owners, int tag = static_cast<int>(dolfinx::MPI::tag::consensus_nbx))#

Create an overlapping (ghosted) index map.

Uses a consensus algorithm to determine ranks that ghost entries owned by the caller. Use the explicit source/destination constructor when these ranks are known.

For example, a two-rank map with one ghost on each rank has layout: Here, | separates owned and ghost entries. 

rank 0: local [0, 1] | [2] -> global [0, 1] | [2] (owner 1)
rank 1: local [0, 1] | [2] -> global [2, 3] | [1] (owner 0)

Note

Collective

Note

Use a distinct tag for overlapping consensus calls. All ranks in one collective call must use the same tag. An MPI barrier before and after the call is an alternative.

Parameters:

  • comm – [in] Communicator that the index map is distributed across.

  • local_size – [in] Number of owned entries. Must be non-negative.

  • ghosts – [in] Unique global indices of ghost entries.

  • owners – [in] Non-self rank (on comm) that owns each entry in ghosts.

  • tag – [in] Tag used in non-blocking MPI calls in the consensus algorithm.

Throws:

std::invalid_argument – If local_size is negative, if ghosts and owners differ in length, or if another ghost data precondition is violated in a Developer build.

Pre:

Every rank in this collective call satisfies the locally checked preconditions below. A violation on only some ranks may deadlock in Release builds.

Pre:

local_size is non-negative and ghosts and owners have equal length; these are always checked locally (no MPI communication). Ghosts must also be unique and non-negative, owners must be valid non-self ranks, and each ghost must be globally owned by its declared rank; these further conditions are checked in Developer builds only, and callers must ensure them in Release builds.

IndexMap(MPI_Comm comm, std::int32_t local_size, const std::array<std::vector<int>, 2> &src_dest, std::span<const std::int64_t> ghosts, std::span<const int> owners)#

Create an overlapping (ghosted) index map.

Use this constructor when source ranks (owners of the caller’s ghosts) and destination ranks (ranks ghosting the caller’s entries) are known.

For example, a two-rank map with one ghost on each rank has layout: Here, | separates owned and ghost entries. 

rank 0: local [0, 1] | [2] -> global [0, 1] | [2] (owner 1)
rank 1: local [0, 1] | [2] -> global [2, 3] | [1] (owner 0)
Rank 0 has source rank 1 and destination rank 1, and conversely for rank 1.

Note

Collective

Parameters:

  • comm – [in] Communicator that the index map is distributed across.

  • local_size – [in] Number of owned entries. Must be non-negative.

  • src_dest – [in] Lists of (0) source and (1) destination ranks. Both lists must be sorted, unique and contain valid ranks. Source ranks must be exactly the unique owners of ghosts; destination ranks must be the ranks that ghost entries owned by the caller.

  • ghosts – [in] Unique global indices of ghost entries.

  • owners – [in] Non-self rank (on comm) that owns each entry in ghosts.

Throws:

std::invalid_argument – If local_size is negative, if ghosts and owners differ in length, or if another ghost data precondition is violated in a Developer build.

Pre:

Every rank in this collective call satisfies the locally checked preconditions below. A violation on only some ranks may deadlock in Release builds.

Pre:

local_size is non-negative and ghosts and owners have equal length; these are always checked locally (no MPI communication). Ghosts must also be unique and non-negative, owners must be valid non-self ranks, and each ghost must be globally owned by its declared rank. For every pair of ranks (a, b), b must be in a’s source list if and only if a is in b’s destination list. These further conditions are checked in Developer builds only, and callers must ensure them in Release builds.

IndexMap(IndexMap &&map) = default#

Move constructor.

~IndexMap() = default#

Destructor.

IndexMap &operator=(IndexMap &&map) = default#

Move assignment.

std::array<std::int64_t, 2> local_range() const noexcept#

Return the global range of owned indices.

std::int32_t num_ghosts() const noexcept#

Return the number of ghost indices.

std::int32_t size_local() const noexcept#

Return the number of owned indices.

std::int64_t size_global() const noexcept#

Return the total number of indices across the communicator.

std::span<const std::int64_t> ghosts() const noexcept#

Return global indices of ghosts in local ghost-index order.

MPI_Comm comm() const#

Return the communicator that the map is defined on.

Returns:

Communicator

void local_to_global(std::span<const std::int32_t> local, std::span<std::int64_t> global) const#

Compute global indices for local indices.

Parameters:

  • local – [in] Local indices in [0, size_local() + num_ghosts()).

  • global – [out] Global indices. Must have at least the size of local.

Throws:

  • std::invalid_argument – If global is smaller than local.

  • std::out_of_range – If the local precondition is violated in a Developer build.

Pre:

local is in range. This condition is checked in Developer builds; callers must ensure it in Release builds.

void global_to_local(std::span<const std::int64_t> global, std::span<std::int32_t> local) const#

Compute local indices for global indices.

Parameters:

  • global – [in] Global indices.

  • local – [out] Local indices. Must have the same size as global. Entries without a local index are set to -1.

Throws:

std::invalid_argument – If global and local differ in size.

std::vector<std::int64_t> global_indices() const#

Return global indices for all local entries, including ghosts.

inline std::span<const int> owners() const noexcept#

Return ranks that own ghost entries.

Returns:

Owner ranks aligned with ghosts().

std::pair<std::vector<int>, std::vector<std::int32_t>> index_to_dest_ranks() const#

Compute sharing ranks for each local index.

Communicates over neighbourhoods built from src() and dest().

Note

Collective

Returns:

(0) Sharing-rank data and (1) offsets. Ranks sharing local index i occupy [offsets[i], offsets[i + 1]). For an owned index, these are the ranks that ghost it; for a ghost index, its owner and the other ranks that ghost it. The caller is excluded.

std::vector<std::int32_t> shared_indices() const#

Return owned indices ghosted by another rank.

Note

Collective

Returns:

Sorted unique local indices.

std::span<const int> src() const noexcept#

Return sorted unique ranks that own the caller’s ghosts.

std::span<const int> dest() const noexcept#

Return sorted unique ranks that ghost entries owned by the caller.

std::vector<std::int32_t> weights_src() const#

Count the caller’s ghosts owned by each source rank.

The returned vector is aligned with src(). Each value is the number of local ghost entries whose owner is the corresponding source rank.

Returns:

Number of ghosts owned by each source rank.

std::vector<std::int32_t> weights_dest() const#

Count the caller’s owned entries ghosted by each destination rank.

The returned vector is aligned with dest(). Each value is the number of local owned entries ghosted by the corresponding destination rank.

Note

Collective

Returns:

Number of entries ghosted by each destination rank.

std::array<std::vector<int>, 2> rank_type(int split_type) const#

Return neighbours in the caller’s MPI split-type group.

Forms a communicator with MPI_Comm_split_type and restricts the caller’s destination and source ranks to the resulting group. The destination ranks ghost entries owned by the caller; the source ranks own ghosts held by the caller. For example, with MPI_COMM_TYPE_SHARED, this identifies neighbouring ranks on the same shared-memory domain.

Note

Collective on comm().

Parameters:

split_type – [in] MPI split type passed to MPI_Comm_split_type, e.g. MPI_COMM_TYPE_SHARED.

Returns:

(0) Destination ranks and (1) source ranks in the split-type group. Ranks are numbered on comm(), rather than on the split communicator.

template<class Container = std::vector<std::int32_t>>
class Scatterer#
#include <Scatterer.h>

A Scatterer supports the scattering and gathering of distributed data that is associated with a common::IndexMap, using MPI.

Scatter and gather operations use MPI neighbourhood collectives.

The implementation is designed for sparse communication patterns, as is typical of patterns based on an IndexMap.

A Scatterer is stateless, i.e. it provides the required information and static data for a given parallel communication pattern but does not provide any communication caches or track the status of MPI requests. Callers of a Scatterer’s members are responsible for managing buffer and MPI request handles.

Creating, copying and destroying a Scatterer are collective, since they create, duplicate and free MPI communicators. Move construction is not collective, but move assignment is, since it frees the communicators held by the assignment target.

A forward scatter sends data associated with owned/local indices to the ranks that ghost them; a reverse scatter sends ghost data back to the owning ranks, to be accumulated into the owned data. Both use the same two-step begin/end pattern, splitting the non-blocking MPI call from completion so that unrelated work can be done while communication is in flight. A round trip for a forward scatter with block size 1, where x holds the owned data and x_ghost the ghost data: 

Scatterer sc(map);
std::vector<std::int64_t> send_buffer(sc.local_indices_block().size());
{
auto& idx = sc.local_indices_block();
for (std::size_t i = 0; i < idx.size(); ++i)
    send_buffer[i] = x[idx[i]];
}
std::vector<std::int64_t> recv_buffer(sc.remote_indices_block().size());
MPI_Request request = MPI_REQUEST_NULL;
sc.scatter_fwd_begin(send_buffer.data(), recv_buffer.data(), 1, request);
// ... unrelated work can be done here while communication is
// in flight, but send_buffer/recv_buffer must not be touched ...
sc.scatter_fwd_end(request);
{
auto& idx = sc.remote_indices_block();
for (std::size_t i = 0; i < idx.size(); ++i)
    x_ghost[idx[i]] = recv_buffer[i];
}
A reverse scatter follows the same pattern with the roles of ::local_indices_block and ::remote_indices_block, and of send_buffer and recv_buffer, swapped; see ::scatter_rev_begin and ::scatter_rev_end.

Template Parameters:

Container – Container type for storing the ‘local’ and ‘remote’ indices. On CPUs this is normally std::vector<std::int32_t>. For GPUs the container should store the indices on the device, e.g. using thrust::device_vector<std::int32_t>.

Public Types

using container_type = Container#

Container type used to store local and remote indices.

Public Functions

inline explicit Scatterer(const IndexMap &map)#

Create a scatterer for data with a layout described by an IndexMap.

Note

Collective.

Parameters:

map – [in] Index map that describes the parallel layout of data.

template<class U>
inline Scatterer(const Scatterer<U> &s)#

Cast-copy constructor.

Create a copy of a Scatterer, where the copy uses a different storage container for indices that are used in MPI communication. Example usage includes creating from a CPU-suitable Scatterer a GPU-suitable Scatterer that can be used with GPU-aware MPI to move data between devices. This would be typical when copying a la::Vector or la::MatrixCSR to/from a GPU. When copying a vector or matrix to/from a GPU, the underlying Scatter that manages parallel communication will usually be copied too with a different storage container.

Note

Collective. The neighbourhood communicators are duplicated, so all ranks must make the copy together.

Parameters:

s – Scatterer to copy

Scatterer(const Scatterer &scatterer) = default#

Copy constructor

Note

Collective, as for the cast-copy constructor. Move instead where the original is no longer required.

Scatterer(Scatterer &&scatterer) = default#

Move constructor

Note

Not collective, unlike the copy constructors: the communicators are taken over rather than duplicated.

~Scatterer() = default#

Destructor

Note

Collective, since the communicators are freed.

Scatterer &operator=(Scatterer &&scatterer) = default#

Move assignment

Note

Collective if this Scatterer holds communicators, since assigning to it frees them.

template<typename T>
inline void scatter_fwd_begin_dtype(const T *send_buffer, T *recv_buffer, MPI_Datatype type, MPI_Request &request) const#

Start a non-blocking neighbourhood collective exchange of owned data with the ranks that ghost it.

The communication is completed by calling Scatterer::scatter_fwd_end. See ::local_indices_block for how to pack send_buffer and ::remote_indices_block for how to unpack recv_buffer.

This is a differently named function rather than an overload of ::scatter_fwd_begin because the underlying type of MPI_Datatype is implementation-defined, and is an integer type in some MPI implementations, so overloading on int and MPI_Datatype is not portably unambiguous.

Note

Collective MPI operation. Every rank in the communicator must call this function, including ranks without neighbours.

Note

The send and receive buffers must not be changed or accessed until after a call to Scatterer::scatter_fwd_end.

Note

The pointers send_buffer and recv_buffer must be pointers to the data on the target device. E.g., if the send and receive buffers are allocated on a GPU, the send_buffer and recv_buffer should be device pointers.

Parameters:

  • send_buffer – [in] Packed local data associated with each owned local index to be sent to processes where the data is ghosted. See Scatterer::local_indices_block for the order of the buffer and how to pack.

  • recv_buffer – [inout] Buffer for storing received data. See Scatterer::remote_indices_block for the order of the buffer and how to unpack.

  • type – [in] MPI datatype for the data associated with one index, e.g. dolfinx::MPI::Datatype<T>(bs).type(). Buffer counts and displacements are in units of type, and the same type must be used on all ranks. MPI keeps a datatype alive until communication using it has completed, so type may be freed as soon as this function returns.

  • request – [out] Handle for tracking the status of the non-blocking communication. Any value passed in is overwritten, and MPI_REQUEST_NULL is returned when this rank has nothing to communicate. The same handle must be passed to Scatterer::scatter_fwd_end to complete the communication.

template<typename T>
inline void scatter_fwd_begin(const T *send_buffer, T *recv_buffer, int bs, MPI_Request &request) const#

Start a non-blocking neighbourhood collective exchange of owned data with the ranks that ghost it.

As ::scatter_fwd_begin_dtype, but with the MPI datatype built from a block size.

Parameters:

  • send_buffer – [in] Packed local data associated with each owned local index to be sent to processes where the data is ghosted. See Scatterer::local_indices_block for the order of the buffer and how to pack.

  • recv_buffer – [inout] Buffer for storing received data. See Scatterer::remote_indices_block for the order of the buffer and how to unpack.

  • bs – [in] Number of values per index map index (the block size). The buffers hold bs values for each index in ::local_indices_block and ::remote_indices_block respectively.

  • request – [out] Handle for tracking the status of the non-blocking communication. Any value passed in is overwritten, and MPI_REQUEST_NULL is returned when this rank has nothing to communicate. The same handle must be passed to Scatterer::scatter_fwd_end to complete the communication.

inline void scatter_fwd_end(MPI_Request &request) const#

Complete a non-blocking MPI neighbourhood collective send.

This function completes the communication started by ::scatter_fwd_begin or ::scatter_fwd_begin_dtype.

Note

Local completion of the caller’s own request, not itself collective. Every rank that called ::scatter_fwd_begin must still call this before reusing the buffers.

Parameters:

request – [inout] Handle returned by the matching begin call. Set to MPI_REQUEST_NULL once the communication has completed.

template<typename T>
inline void scatter_rev_begin_dtype(const T *send_buffer, T *recv_buffer, MPI_Datatype type, MPI_Request &request) const#

Start a non-blocking neighbourhood collective exchange of ghost data with the owning ranks.

The communication is completed by calling Scatterer::scatter_rev_end. See ::remote_indices_block for how to pack send_buffer and ::local_indices_block for how to unpack recv_buffer.

This is a differently named function rather than an overload of ::scatter_rev_begin because the underlying type of MPI_Datatype is implementation-defined, and is an integer type in some MPI implementations, so overloading on int and MPI_Datatype is not portably unambiguous.

Note

Collective MPI operation. Every rank in the communicator must call this function, including ranks without neighbours.

Note

The send and receive buffers must not be changed or accessed until after a call to Scatterer::scatter_rev_end.

Note

The pointers send_buffer and recv_buffer must be pointers to the data on the target device. E.g., if the send and receive buffers are allocated on a GPU, the send_buffer and recv_buffer should be device pointers.

Parameters:

  • send_buffer – [in] Data associated with each ghost index. This data is sent to the process that owns the index. See Scatterer::remote_indices_block for the order of the buffer and how to pack.

  • recv_buffer – [inout] Buffer for storing received data. See Scatterer::local_indices_block for the order of the buffer and how to unpack.

  • type – [in] MPI datatype for the data associated with one index, e.g. dolfinx::MPI::Datatype<T>(bs).type(). Buffer counts and displacements are in units of type, and the same type must be used on all ranks. MPI keeps a datatype alive until communication using it has completed, so type may be freed as soon as this function returns.

  • request – [out] Handle for tracking the status of the non-blocking communication. Any value passed in is overwritten, and MPI_REQUEST_NULL is returned when this rank has nothing to communicate. The same handle must be passed to Scatterer::scatter_rev_end to complete the communication.

template<typename T>
inline void scatter_rev_begin(const T *send_buffer, T *recv_buffer, int bs, MPI_Request &request) const#

Start a non-blocking neighbourhood collective exchange of ghost data with the owning ranks.

As ::scatter_rev_begin_dtype, but with the MPI datatype built from a block size.

Parameters:

  • send_buffer – [in] Data associated with each ghost index. This data is sent to the process that owns the index. See Scatterer::remote_indices_block for the order of the buffer and how to pack.

  • recv_buffer – [inout] Buffer for storing received data. See Scatterer::local_indices_block for the order of the buffer and how to unpack.

  • bs – [in] Number of values per index map index (the block size). The buffers hold bs values for each index in ::remote_indices_block and ::local_indices_block respectively.

  • request – [out] Handle for tracking the status of the non-blocking communication. Any value passed in is overwritten, and MPI_REQUEST_NULL is returned when this rank has nothing to communicate. The same handle must be passed to Scatterer::scatter_rev_end to complete the communication.

inline void scatter_rev_end(MPI_Request &request) const#

Complete a non-blocking MPI neighbourhood collective send.

This function completes the communication started by ::scatter_rev_begin or ::scatter_rev_begin_dtype.

Note

Local completion of the caller’s own request, not itself collective. Every rank that called ::scatter_rev_begin must still call this before reusing the buffers.

Parameters:

request – [inout] Handle returned by the matching begin call. Set to MPI_REQUEST_NULL once the communication has completed.

inline const container_type &local_indices_block() const noexcept#

Array of indices for packing/unpacking owned data to/from a send/receive buffer.

For a forward scatter, the indices are used to copy required entries in the owned part of the data array into the appropriate position in a send buffer. For a reverse scatter, indices are used for assigning (accumulating) the receive buffer values into the correct position in the owned part of the data array.

The indices are in blocks, so for a block size bs a buffer holds bs values per index and must be bs * local_indices_block().size() long.

For a forward scatter, if x is the owned part of an array and send_buffer is the send buffer, send_buffer is packed such that: 

auto& idx = scatterer.local_indices_block()
std::vector<T> send_buffer(bs * idx.size())
for (std::size_t i = 0; i < idx.size(); ++i)
    for (int j = 0; j < bs; ++j)
        send_buffer[i * bs + j] = x[idx[i] * bs + j];

For a reverse scatter, if recv_buffer is the received buffer, then x is updated by 

auto& idx = scatterer.local_indices_block()
std::vector<T> recv_buffer(bs * idx.size())
for (std::size_t i = 0; i < idx.size(); ++i)
    for (int j = 0; j < bs; ++j)
        x[idx[i] * bs + j]
            = op(recv_buffer[i * bs + j], x[idx[i] * bs + j]);

where op is a binary operation, e.g. x[...] = buffer[...] or x[...] += buffer[...].

Returns:

Indices container.

inline const container_type &remote_indices_block() const noexcept#

Array of indices for packing/unpacking ghost data to/from a send/receive buffer.

For a forward scatter, the indices are used to unpack received data into ghost entries. For a reverse scatter, indices are used to pack ghost entries into the send buffer.

The indices are in blocks, so for a block size bs a buffer holds bs values per index and must be bs * remote_indices_block().size() long.

For a forward scatter, if xg is the ghost part of the data array and recv_buffer is the receive buffer, xg is updated as 

auto& idx = scatterer.remote_indices_block()
std::vector<T> recv_buffer(bs * idx.size())
for (std::size_t i = 0; i < idx.size(); ++i)
    for (int j = 0; j < bs; ++j)
        xg[idx[i] * bs + j] = recv_buffer[i * bs + j];

For a reverse scatter, if send_buffer is the send buffer, then send_buffer is packed such that: 

auto& idx = scatterer.remote_indices_block()
std::vector<T> send_buffer(bs * idx.size())
for (std::size_t i = 0; i < idx.size(); ++i)
    for (int j = 0; j < bs; ++j)
        send_buffer[i * bs + j] = xg[idx[i] * bs + j];

Returns:

Block indices container.

class TimeLogger#
#include <TimeLogger.h>

Time logger maintaining data collected by Timer, if registered.

Note

This is a monotstate, i.e. the data members are static and thus timings are aggregated into a single map.

Public Functions

void register_timing(std::string_view task, std::chrono::duration<double, std::ratio<1>> wall)#

Register timing (for later summary).

Table timing_table() const#

Return a summary of timings and tasks in a Table.

void list_timings(MPI_Comm comm, Table::Reduction reduction) const#

List a summary of timings and tasks. Reduction type is printed.

Parameters:

  • comm – MPI Communicator

  • reduction – Reduction type (min, max or average)

std::pair<int, std::chrono::duration<double, std::ratio<1>>> timing(std::string_view task) const#

Return timing.

Parameters:

task – [in] The task name to retrieve the timing for

Returns:

Values (count, total wall time) for given task.

std::map<std::string, std::pair<int, std::chrono::duration<double, std::ratio<1>>>, std::less<>> timings() const#

Logged elapsed times.

Returns:

Elapsed [task id: (count, total wall time)].

Public Static Functions

static TimeLogger &instance()#

Singleton access.

Returns:

Unique time logger object.

template<typename T = std::chrono::high_resolution_clock>
class Timer#
#include <Timer.h>

Timer for measuring and logging elapsed time durations.

The basic usage is 

Timer timer("Assembling over cells");
The timer is started at construction and timing ends when the timer is destroyed (goes out-of-scope). The timer can be started (reset) and stopped explicitly by 
timer.start();
 /* .... */
timer.stop();
A summary of registered elapsed times can be printed by calling: 
list_timings();
Registered elapsed times are logged when (1) the timer goes out-of-scope or (2) Timer::flush() is called.

Public Functions

inline Timer(std::optional<std::string> task = std::nullopt)#

Create and start timer.

Elapsed time is optionally registered in the logger when the Timer destructor is called.

Parameters:

task – [in] Name used to registered the elapsed time in the logger. If no name is set, the elapsed time is not registered in the logger.

inline Timer(Timer &&timer) noexcept#

Move constructor.

Note

Resets the moved-from timer explicitly. A moved-from std::optional stays engaged, so a defaulted move would leave the source registering an empty task name.

inline ~Timer()#

If timer is still running, it is stopped. Elapsed time is registered in the logger.

inline Timer &operator=(Timer &&timer) noexcept#

Move assignment.

Note

Any elapsed time not yet registered by the target is discarded. See the move constructor for why the source is reset.

inline void start()#

Reset elapsed time and (re-)start timer.

template<typename Period = std::ratio<1>>
inline std::chrono::duration<double, Period> elapsed() const#

Elapsed time since time has been started.

Default duration unit is seconds.

Returns:

Elapsed time duration.

template<typename Period = std::ratio<1>>
inline std::chrono::duration<double, Period> stop()#

Stop timer and return elapsed time.

Default duration unit is seconds.

Returns:

Elapsed time duration.

inline void resume()#

Resume a stopped timer.

Does nothing if timer has not been stopped.

inline void flush()#

Flush timer duration to the logger.

An instance of a timer can be flushed to the logger only once. Subsequent calls will have no effect and will not trigger any logging.

Pre:

Timer must have been stopped before flushing.

namespace petsc#

PETSc error handling helpers shared across DOLFINx’s PETSc wrappers.

Functions

void error(PetscErrorCode error_code, std::string_view petsc_function, std::source_location loc = std::source_location::current())#

Print error message for a PETSc call that returned an error and throw a std::runtime_error.

Parameters:

  • error_code – [in] PETSc error code

  • petsc_function – [in] Name of the PETSc function that returned error_code

  • loc – [in] Call site of the failed PETSc call (captured automatically; do not pass explicitly)

inline void check(PetscErrorCode ierr, std::string_view petsc_function, std::source_location loc = std::source_location::current())#

Throw a std::runtime_error via error() if ierr indicates a PETSc call failed.

Parameters:

  • ierr – [in] PETSc error code returned by petsc_function

  • petsc_function – [in] Name of the PETSc function that returned ierr

  • loc – [in] Call site of the failed PETSc call (captured automatically; do not pass explicitly)

void set_option(std::string option)#

Set a PETSc option in the PETSc options/parameter database. The option must not be prefixed by ‘-’, e.g. 

common::petsc::set_option("mat_mumps_icntl_14", 40);
template<typename T>
void set_option(std::string option, const T &value)#

Set a PETSc option that takes a value in the PETSc options/parameter database. The option must not be prefixed by ‘-’, e.g. 

common::petsc::set_option("mat_mumps_icntl_14", 40);
void clear_option(std::string option)#

Clear a PETSc option from the PETSc options/parameter database.

void clear_options()#

Clear the PETSc global options/parameter database.

MPI#

namespace MPI#

MPI support functionality.

Enums

enum class tag : int#

MPI communication tags.

Values:

enumerator consensus_pcx#
enumerator consensus_nbx#

Functions

int rank(MPI_Comm comm)#

Return process rank for the communicator.

int size(MPI_Comm comm)#

Return size of the group (number of processes) associated with the communicator.

void check_error(MPI_Comm comm, int code) noexcept#

Check MPI error code. If the error code is not equal to MPI_SUCCESS, then std::abort is called.

Note

Aborts rather than throwing: MPI’s state is undefined after an error, so there is nothing to recover into. Being noexcept makes this safe to call from a destructor.

Parameters:

  • comm – [in] MPI communicator.

  • code – [in] Error code returned by an MPI function call.

constexpr int index_owner(int size, std::size_t index, std::size_t N)#

Return which rank owns index in global range [0, N - 1] (inverse of MPI::local_range).

Parameters:

  • size – [in] Number of MPI ranks.

  • index – [in] The index to determine the owning rank of.

  • N – [in] Total number of indices.

Returns:

Rank of the owning process.

std::vector<int> compute_graph_edges_pcx(MPI_Comm comm, std::span<const int> edges)#

Determine incoming graph edges using the PCX consensus algorithm.

Given a list of outgoing edges (destination ranks) from this rank, this function returns the incoming edges (source ranks) to this rank.

Note

This function is for sparse communication patterns, i.e. where the number of ranks that communicate with each other is relatively small. It is not scalable as arrays the size of the communicator are allocated. It implements the PCX algorithm described in https://dx.doi.org/10.1145/1837853.1693476.

Note

For sparse graphs, this function has \(O(p)\) cost, where \(p\)is the number of MPI ranks. It is suitable for modest MPI rank counts.

Note

The order of the returned ranks is not deterministic.

Note

Collective

Parameters:

  • comm – [in] MPI communicator

  • edges – [in] Edges (ranks) from this rank (the caller).

Pre:

edges must not contain duplicate ranks (a duplicate can leave a message unconsumed on the shared PCX tag, corrupting a later call).

Returns:

Ranks that have defined edges from them to this rank.

std::vector<int> compute_graph_edges_nbx(MPI_Comm comm, std::span<const int> edges, int tag = static_cast<int>(tag::consensus_nbx))#

Determine incoming graph edges using the NBX consensus algorithm.

Given a list of outgoing edges (destination ranks) from this rank, this function returns the incoming edges (source ranks) to this rank.

Note

This function is for sparse communication patterns, i.e. where the number of ranks that communicate with each other is relatively small. It is scalable, i.e. no arrays the size of the communicator are constructed and the communication pattern is sparse. It implements the NBX algorithm presented in https://dx.doi.org/10.1145/1837853.1693476.

Note

The order of the returned ranks is not deterministic.

Note

Collective.

Note

An alternative to passing a tag is to ensure that there is an implicit or explicit barrier before and after the call to this function.

Parameters:

  • comm – [in] MPI communicator

  • edges – [in] Edges (ranks) from this rank (the caller).

  • tag – [in] Tag used in non-blocking MPI calls. A tag can be required when this function is called a second time on some ranks before a previous call has completed on all other ranks.

Pre:

edges must not contain duplicate ranks.

Returns:

Ranks that have defined edges from them to this rank.

std::pair<std::vector<int>, std::vector<int>> compute_graph_edges_nbx(MPI_Comm comm, std::span<const int> edges0, int tag0, std::span<const int> edges1, int tag1)#

Determine incoming graph edges for two independent edge sets using a single, overlapped NBX consensus round.

Equivalent to calling MPI::compute_graph_edges_nbx independently for edges0 and edges1, but runs both consensus rounds concurrently behind a single shared barrier rather than one after the other, halving the number of global barrier round-trips when the two edge sets can be computed without waiting on one another.

Note

tag0 and tag1 must differ from one another, and from any tag used by another consensus round that may be in flight concurrently on comm.

Note

Collective.

Parameters:

  • comm – [in] MPI communicator

  • edges0 – [in] Edges (ranks) from this rank (the caller) for the first edge set.

  • tag0 – [in] Tag used for edges0’s messages.

  • edges1 – [in] Edges (ranks) from this rank (the caller) for the second edge set.

  • tag1 – [in] Tag used for edges1’s messages.

Returns:

Ranks with defined edges to this rank, for (0) edges0 and (1) edges1.

template<std::ranges::contiguous_range U>
std::pair<std::vector<std::int32_t>, std::vector<std::ranges::range_value_t<U>>> distribute_to_postoffice(MPI_Comm comm, const U &x, std::array<std::int64_t, 2> shape, std::int64_t rank_offset)#

Send row data to its ‘post office’ rank.

x is a contiguous local block of a larger row-major array distributed over comm, with local row i at global index rank_offset + i. Each row is sent to its post office rank (dolfinx::MPI::index_owner applied to the row’s global index and shape[0]), except rows for which the caller is itself the post office, which are left in place.

Parameters:

  • comm – [in] MPI communicator.

  • x – [in] Local block of the array to distribute (row-major).

  • shape – [in] Global shape of the array, {num_rows, num_cols}.

  • rank_offset – [in] Global row index of local row 0 in x, usually computed with MPI_Exscan.

Returns:

(0) position of each received row within the caller’s post-office partition of [0, shape[0]), and (1) the received row data (row-major). Rows for which the caller is itself the post office are not included.

template<std::ranges::contiguous_range U>
std::vector<std::ranges::range_value_t<U>> distribute_from_postoffice(MPI_Comm comm, std::span<const std::int64_t> indices, const U &x, std::array<std::int64_t, 2> shape, std::int64_t rank_offset)#

Fetch rows of a distributed row-major array via their post office ranks.

x is a contiguous local block of a larger row-major array distributed over comm, with local row i at global index rank_offset + i. For each global row index in indices (which may contain repeats), returns that row – read directly from x if already local, otherwise requested from its post office rank via MPI neighbourhood collectives. Scalable provided each rank exchanges with only a modest number of others.

Parameters:

  • comm – [in] MPI communicator.

  • indices – [in] Global row indices required by the caller.

  • x – [in] Local block of the array to distribute (row-major).

  • shape – [in] Global shape of the array, {num_rows, num_cols}.

  • rank_offset – [in] Global row index of local row 0 in x (usually an exclusive scan of row counts over the communicator x is distributed across, which may differ from comm).

Returns:

The row for each entry of indices, in the same order (row-major storage).

Pre:

shape[1] > 0.

template<std::ranges::contiguous_range U>
std::vector<std::ranges::range_value_t<U>> distribute_data(MPI_Comm comm0, std::span<const std::int64_t> indices, MPI_Comm comm1, const U &x, int shape1)#

Distribute rows of a row-major array to the ranks that require them, via the post office pattern.

Scalable provided each rank exchanges with only a modest number of others.

Parameters:

  • comm0 – [in] Communicator over which indices are resolved and the result is returned.

  • indices – [in] Global row indices required by the calling rank.

  • comm1 – [in] Communicator across which x is distributed – typically comm0 itself or a sub-communicator of it, and MPI_COMM_NULL on ranks where x is empty.

  • x – [in] Local block of rows to distribute (row-major); local row i has global index given by an exclusive scan of local row counts over comm1.

  • shape1 – [in] Number of columns of x.

Returns:

The row for each entry of indices, in the same order (row-major storage).

Pre:

shape1 > 0.

Variables

template<typename T>
MPI_Datatype mpi_t = mpi_datatype<T>()#

Retrieves the MPI data type associated to the provided type.

Template Parameters:

T – cpp type to map

class Comm#
#include <MPI.h>

A duplicate MPI communicator and manage lifetime of the communicator.

Public Functions

explicit Comm(MPI_Comm comm, bool duplicate = true)#

Duplicate communicator and wrap duplicate.

Comm(const Comm &comm) noexcept#

Copy constructor.

Comm(Comm &&comm) noexcept#

Move constructor.

~Comm()#

Destructor (frees wrapped communicator).

Comm &operator=(Comm &&comm) noexcept#

Move assignment operator.

MPI_Comm comm() const noexcept#

Return the underlying MPI_Comm object.

template<typename T>
class Datatype#
#include <MPI.h>

An MPI datatype for count contiguous values of type T, and manage its lifetime.

Lets a buffer be sent with counts and displacements measured in groups of count values rather than in single values. The group is an index map block size in common::Scatterer, and the row width of a row-major buffer elsewhere; it is not required to be either.

For count == 1 no datatype is created and mpi_t<T> is used directly, so this is cheap to construct in that case.

Note

MPI keeps a datatype alive until communication using it has completed, so this may be destroyed as soon as a non-blocking call using it has been started.

Template Parameters:

T – Type of each value.

Public Functions

inline explicit Datatype(int count)#

Create a datatype for count contiguous values.

Parameters:

count – [in] Number of values MPI should treat as one unit.

inline Datatype(Datatype &&type) noexcept#

Move constructor.

inline ~Datatype()#

Destructor (frees the datatype, if one was created).

inline Datatype &operator=(Datatype &&type) noexcept#

Move assignment.

inline MPI_Datatype type() const noexcept#

The datatype to pass to MPI.

Returns:

Contiguous type, or mpi_t<T> when count is one.

class Request#
#include <MPI.h>

Holder for the request of a non-blocking operation.

MPI_Request is a plain handle, so a class storing one bare cannot use a defaulted move: the moved-from object would be left naming the same request as the target. This holder transfers the request on move and leaves the source null, which keeps the owner’s own move operations = default and correct as members are added.

Copying does not transfer a request in flight – the new request is null – so a copied owner does not wait on an operation it did not start. The owner’s copy operations therefore stay usable.

Note

Nothing is released on destruction. A request is consumed by the MPI_Wait that completes it, which the owner is responsible for.

Public Functions

Request() = default#

Create a null request.

inline Request(const Request&) noexcept#

Copy constructor. The copy has no request in flight.

inline Request(Request &&request) noexcept#

Move constructor.

~Request() = default#

Destructor.

inline Request &operator=(const Request&) noexcept#

Copy assignment. Leaves this with no request in flight.

inline Request &operator=(Request &&request) noexcept#

Move assignment.

inline MPI_Request &request() noexcept#

The handle to pass to MPI.

Returns:

Request, MPI_REQUEST_NULL if no operation is in flight.

inline const MPI_Request &request() const noexcept#

The handle.

Returns:

Request, MPI_REQUEST_NULL if no operation is in flight.

namespace impl#

Functions

std::tuple<std::vector<int>, std::vector<std::int32_t>, std::vector<std::int32_t>> postoffice_plan(int size, int rank, std::int32_t shape0_local, std::int64_t shape0, std::int64_t rank_offset)#

Local (non-communicating) part of distribute_to_postoffice.

For each local row of x, determines its post office (destination) rank by applying dolfinx::MPI::index_owner to the row’s global index and shape0, skipping rows already owned by this rank (which never need to be sent anywhere). The remaining (destination, position) pairs are grouped by destination with a radix sort – this list can have hundreds of thousands of entries for a large mesh/problem, too many for a generic comparison sort – producing the compact per-neighbour form (unique destination ranks, row count per destination, and a local-row-to-neighbour-rank map) that postoffice_exchange needs to drive MPI neighbourhood collectives without a further per-row lookup during packing.

Factored out of distribute_to_postoffice so distribute_from_postoffice can reuse the same planning step when it overlaps this push with the NBX round that resolves who is requesting data from it.

Parameters:

  • size – [in] Number of ranks in the communicator x is distributed/required across.

  • rank – [in] Rank of the caller in that communicator.

  • shape0_local – [in] Number of local rows of x (before any post-office redistribution).

  • shape0 – [in] Global number of rows (row count summed over all ranks).

  • rank_offset – [in] Global row index of local row 0 in x.

Returns:

(0) unique destination ranks, (1) number of rows destined for each rank in (0), and (2) a map from local row index to position in (0), or -1 if the row is already owned by this rank.

template<std::ranges::contiguous_range U>
std::pair<std::vector<std::int32_t>, std::vector<std::ranges::range_value_t<U>>> postoffice_exchange(MPI_Comm comm, const U &x, std::array<std::int64_t, 2> shape, std::int64_t rank_offset, std::span<const int> dest, std::span<const std::int32_t> num_items_per_dest0, std::span<const std::int32_t> pos_to_neigh_rank, std::span<const int> src)#

Neighbourhood data exchange step of distribute_to_postoffice, given an already-resolved src (e.g. from a single or overlapped NBX consensus round).

Parameters:

  • comm – [in] MPI communicator.

  • x – [in] Local block of the array to distribute (row-major).

  • shape – [in] Global shape of the array, {num_rows, num_cols}.

  • rank_offset – [in] Global row index of local row 0 in x.

  • dest – [in] Destination (post office) ranks, from postoffice_plan.

  • num_items_per_dest0 – [in] Number of rows destined for each rank in dest, from postoffice_plan.

  • pos_to_neigh_rank – [in] Map from local row index to position in dest, from postoffice_plan.

  • src – [in] Ranks that will send this rank data, i.e. the incoming edges resolved from dest (e.g. via compute_graph_edges_nbx).

Returns:

(0) position of each received row within the caller’s post-office partition of [0, shape[0]), and (1) the received row data (row-major).

Tables#

class Table#

This class provides storage and pretty-printing for tables.

Example usage:

Table table(“Timings”); table.set(“Foo”, “Assemble”, 0.010); table.set(“Foo”, “Solve”, 0.020); table.set(“Bar”, “Assemble”, 0.011); table.set(“Bar”, “Solve”, 0.019);

Public Types

enum class Reduction : std::uint8_t#

Types of MPI reduction available for Table, to get the max, min or average values over an MPI_Comm

Values:

enumerator average#
enumerator max#
enumerator min#

Public Functions

Table(std::string title = "", bool right_justify = true)#

Create empty table.

Table(const Table &table) = default#

Copy constructor.

Table(Table &&table) = default#

Move constructor.

~Table() = default#

Destructor.

Table &operator=(const Table &table) = default#

Assignment operator.

Table &operator=(Table &&table) = default#

Move assignment.

void set(std::string_view row, std::string_view col, std::variant<std::string, int, double> value)#

Set table entry

Parameters:

  • row – [in] Row name

  • col – [in] Column name

  • value – [in] The value to set

std::variant<std::string, int, double> get(std::string_view row, std::string_view col) const#

Get value of table entry

Parameters:

  • row – [in] Row name

  • col – [in] Column name

Returns:

Returns the entry for requested row and columns

Table reduce(MPI_Comm comm, Reduction reduction) const#

Do MPI reduction on Table

Parameters:

  • comm – [in] MPI communicator

  • reduction – [in] Type of reduction to perform

Returns:

Reduced Table

std::string str() const#

Return string representation of the table.

Public Members

std::string name#

Table name.

Concepts#

template<class T>
concept scalar#
template<class R>
concept LocalIndexRange#
#include <types.h>

Concept for a contiguous list of process-local indices, e.g. the row or column indices passed to a matrix set function.

On this page
Show Source