Common (`dolfinx::common`)#

namespace common#

Miscellaneous classes, functions and types.

Generic tools.

This namespace provides utility type functions for managing subsystems, convenience classes and library-wide typedefs.

Enums

enum class IndexMapOrder : bool#

Control ghost-index ordering in sub-index maps.

Values:

enumerator preserve#: Preserve the ordering of ghost indices.

enumerator any#: Allow arbitrary ghost-index ordering.

Functions

std::vector<std::int32_t> compute_owned_indices(std::span<const std::int32_t> indices, const IndexMap &map)#

Return selected indices owned by the calling rank.

Includes locally owned entries in indices and entries selected as ghosts on other ranks.

For example, on two ranks, suppose rank 0 owns global indices [0, 1] and has global index 2 as local ghost index 2, while rank 1 owns global indices [2, 3] and has global index 1 as local ghost index 2. If both ranks pass local index [2], the results are [1] on rank 0 and [0] on rank 1: each owns the global index selected as a ghost by the other rank.

Note

Collective

Parameters:

indices – [in] Sorted unique local indices (owned or ghost) in [0, map.size_local() + map.num_ghosts()).
map – [in] The index map.

Throws:

std::invalid_argument – If the indices precondition is violated in a Developer build.

Pre:

indices is sorted, unique, and in range. This condition is checked in Developer builds; callers must ensure it in Release builds.

Returns:

Local indices owned by the calling rank.

std::tuple<std::int64_t, std::vector<std::int32_t>, std::vector<std::vector<std::int64_t>>, std::vector<std::vector<int>>> stack_index_maps(const std::vector<std::pair<std::reference_wrapper<const IndexMap>, int>> &maps)#

Compute layout data for a concatenated index map.

Locally owned entries remain owned by the caller. Ghost entries are grouped by input map in maps.

Note

Collective. Maps with a block size are unrolled.

Parameters:: maps – [in] Non-empty pairs of index maps and positive block sizes. All maps must use the same communicator.
Pre:: All ranks supply corresponding maps in the same order.
Returns:: (0) Global offset on the calling rank, (1) local offsets for owned entries in each map, (2) global ghost indices for each map, and (3) their owner ranks.

std::tuple<IndexMap, std::vector<std::int32_t>, bool> create_sub_index_map(const IndexMap &imap, std::span<const std::int32_t> indices, IndexMapOrder order = IndexMapOrder::any)#

Create an index map from a subset of an existing map.

Note

Collective

Note

(2) is rank-local and is not reduced: it can be true on some ranks and false on others. A caller that requires ownership to be preserved must reduce it (e.g. MPI_Allreduce with MPI_LOR) before acting on it, since throwing on only some ranks would leave the others in a subsequent collective.

Parameters:

imap – [in] Parent map to create a new sub-map from.
indices – [in] Local indices in imap (owned and ghost) to include in the new index map.
order – [in] Control the order in which ghost indices appear in the new map.

Throws:

std::invalid_argument – If the indices precondition is violated in a Developer build.

Pre:

indices contains unique local indices in range. This condition is checked in Developer builds; callers must ensure it in Release builds.

Returns:

(0) New index map, (1) corresponding local indices in imap, and (2) whether any index acquired a new owner in the submap. An index selected only by ghosting ranks, and not by its owner, is given a new owner in the submap; (2) reports whether this happened.

constexpr std::array<std::int64_t, 2> local_range(int index, std::int64_t N, int size)#

Partition a global range [0, N - 1] across callers into non-overlapping sub-partitions of almost equal size. Returns the local partition for the caller. The local partition range.

Partitions [0, N) into size non-overlapping partitions [n_(i0}, n_(i1)), where i is index and n_(i1) == n_((i+1)0).

Parameters:

index – [in] Index of the partition to compute.
N – [in] Global range to partition.
size – [in] Number of partitions into which to partition N.

template<std::ranges::input_range U, std::ranges::input_range V> std::pair<std::vector<typename U::value_type>, std::vector<typename V::value_type>> sort_unique(const U &indices, const V &values)#

Sort two arrays based on the values in array indices.

Any duplicate indices and the corresponding value are removed. In the case of duplicates, the entry with the smallest value is retained.

Parameters:

indices – [in] Array of indices.
values – [in] Array of values.

Returns:

Sorted (indices, values), with sorting based on indices.

template<class T> std::size_t hash_local(const T &x)#

Compute a hash of a given object.

The hash is computed using Boost container hash (https://www.boost.org/doc/libs/release/libs/container_hash/).

Parameters:: x – [in] The object to compute a hash of.
Returns:: The hash values.

template<class T> std::size_t hash_global(MPI_Comm comm, const T &x)#

Compute a hash for a distributed (MPI) object.

A hash is computed on each process for the local part of the object. Then, a hash of the std::vector containing each local hash key in rank order is returned.

Note

Collective

Parameters:

comm – [in] The communicator on which to compute the hash.
x – [in] The object to compute a hash of.

Returns:

The hash values.

class IndexMap#

#include <IndexMap.h>

Distribution of a global index range [0, N) across MPI ranks.

Each rank owns a contiguous global range. Local indices in [0, size_local()) address owned entries; remaining local indices address ghost entries.

Public Functions

IndexMap(MPI_Comm comm, std::int32_t local_size)#

Create a non-overlapping index map.

For example, if rank 0 has local_size = 2 and rank 1 has local_size = 3, the layout is:

rank 0: local [0, 1]    -> global [0, 1]
rank 1: local [0, 1, 2] -> global [2, 3, 4]

Note

Collective

Parameters:

comm – [in] Communicator that the index map is distributed across.
local_size – [in] Number of owned entries. Must be non-negative.

Throws:

std::invalid_argument – If local_size is negative.

Pre:

Every rank in this collective call supplies a non-negative local_size. A violation on only some ranks may deadlock in Release builds.

Pre:

local_size is non-negative; this is always checked locally (no MPI communication).

IndexMap(MPI_Comm comm, std::int32_t local_size, std::span<const std::int64_t> ghosts, std::span<const int> owners, int tag = static_cast<int>(dolfinx::MPI::tag::consensus_nbx))#

Create an overlapping (ghosted) index map.

Uses a consensus algorithm to determine ranks that ghost entries owned by the caller. Use the explicit source/destination constructor when these ranks are known.

For example, a two-rank map with one ghost on each rank has layout: Here, | separates owned and ghost entries.

rank 0: local [0, 1] | [2] -> global [0, 1] | [2] (owner 1)
rank 1: local [0, 1] | [2] -> global [2, 3] | [1] (owner 0)

Note

Collective

Note

Use a distinct tag for overlapping consensus calls. All ranks in one collective call must use the same tag. An MPI barrier before and after the call is an alternative.

Parameters:

comm – [in] Communicator that the index map is distributed across.
local_size – [in] Number of owned entries. Must be non-negative.
ghosts – [in] Unique global indices of ghost entries.
owners – [in] Non-self rank (on comm) that owns each entry in ghosts.
tag – [in] Tag used in non-blocking MPI calls in the consensus algorithm.

Throws:

std::invalid_argument – If local_size is negative, if ghosts and owners differ in length, or if another ghost data precondition is violated in a Developer build.

Pre:

Every rank in this collective call satisfies the locally checked preconditions below. A violation on only some ranks may deadlock in Release builds.

Pre:

local_size is non-negative and ghosts and owners have equal length; these are always checked locally (no MPI communication). Ghosts must also be unique and non-negative, owners must be valid non-self ranks, and each ghost must be globally owned by its declared rank; these further conditions are checked in Developer builds only, and callers must ensure them in Release builds.

IndexMap(MPI_Comm comm, std::int32_t local_size, const std::array<std::vector<int>, 2> &src_dest, std::span<const std::int64_t> ghosts, std::span<const int> owners)#

Create an overlapping (ghosted) index map.

Use this constructor when source ranks (owners of the caller’s ghosts) and destination ranks (ranks ghosting the caller’s entries) are known.

For example, a two-rank map with one ghost on each rank has layout: Here, | separates owned and ghost entries.

rank 0: local [0, 1] | [2] -> global [0, 1] | [2] (owner 1)
rank 1: local [0, 1] | [2] -> global [2, 3] | [1] (owner 0)

Rank 0 has source rank 1 and destination rank 1, and conversely for rank 1.

Note

Collective

Parameters:

comm – [in] Communicator that the index map is distributed across.
local_size – [in] Number of owned entries. Must be non-negative.
src_dest – [in] Lists of (0) source and (1) destination ranks. Both lists must be sorted, unique and contain valid ranks. Source ranks must be exactly the unique owners of ghosts; destination ranks must be the ranks that ghost entries owned by the caller.
ghosts – [in] Unique global indices of ghost entries.
owners – [in] Non-self rank (on comm) that owns each entry in ghosts.

Throws:

std::invalid_argument – If local_size is negative, if ghosts and owners differ in length, or if another ghost data precondition is violated in a Developer build.

Pre:

Every rank in this collective call satisfies the locally checked preconditions below. A violation on only some ranks may deadlock in Release builds.

Pre:

local_size is non-negative and ghosts and owners have equal length; these are always checked locally (no MPI communication). Ghosts must also be unique and non-negative, owners must be valid non-self ranks, and each ghost must be globally owned by its declared rank. For every pair of ranks (a, b), b must be in a’s source list if and only if a is in b’s destination list. These further conditions are checked in Developer builds only, and callers must ensure them in Release builds.

IndexMap(IndexMap &&map) = default#: Move constructor.

~IndexMap() = default#: Destructor.

IndexMap &operator=(IndexMap &&map) = default#: Move assignment.

std::array<std::int64_t, 2> local_range() const noexcept#: Return the global range of owned indices.

std::int32_t num_ghosts() const noexcept#: Return the number of ghost indices.

std::int32_t size_local() const noexcept#: Return the number of owned indices.

std::int64_t size_global() const noexcept#: Return the total number of indices across the communicator.

std::span<const std::int64_t> ghosts() const noexcept#: Return global indices of ghosts in local ghost-index order.

MPI_Comm comm() const#

Return the communicator that the map is defined on.

Returns:: Communicator

void local_to_global(std::span<const std::int32_t> local, std::span<std::int64_t> global) const#

Compute global indices for local indices.

Parameters:

local – [in] Local indices in [0, size_local() + num_ghosts()).
global – [out] Global indices. Must have at least the size of local.

Throws:

std::invalid_argument – If global is smaller than local.
std::out_of_range – If the local precondition is violated in a Developer build.

Pre:

local is in range. This condition is checked in Developer builds; callers must ensure it in Release builds.

void global_to_local(std::span<const std::int64_t> global, std::span<std::int32_t> local) const#

Compute local indices for global indices.

Parameters:

global – [in] Global indices.
local – [out] Local indices. Must have the same size as global. Entries without a local index are set to -1.

Throws:

std::invalid_argument – If global and local differ in size.

std::vector<std::int64_t> global_indices() const#: Return global indices for all local entries, including ghosts.

inline std::span<const int> owners() const noexcept#

Return ranks that own ghost entries.

Returns:: Owner ranks aligned with ghosts().

std::pair<std::vector<int>, std::vector<std::int32_t>> index_to_dest_ranks(int tag = static_cast<int>(dolfinx::MPI::tag::consensus_nbx)) const#

Compute sharing ranks for each local index.

Note

Collective

Note

See IndexMap(MPI_Comm, std::int32_t, std::span<conststd::int64_t>, std::span<const int>, int) for tag requirements.

Parameters:: tag – [in] Tag to pass to MPI calls.
Returns:: (0) Sharing-rank data and (1) offsets. Ranks sharing local index i occupy [offsets[i], offsets[i + 1]).

std::vector<std::int32_t> shared_indices() const#

Return owned indices ghosted by another rank.

Note

Collective

Returns:: Sorted unique local indices.

std::span<const int> src() const noexcept#: Return sorted unique ranks that own the caller’s ghosts.

std::span<const int> dest() const noexcept#: Return sorted unique ranks that ghost entries owned by the caller.

std::vector<std::int32_t> weights_src() const#

Count the caller’s ghosts owned by each source rank.

The returned vector is aligned with src(). Each value is the number of local ghost entries whose owner is the corresponding source rank.

Returns:: Number of ghosts owned by each source rank.

std::vector<std::int32_t> weights_dest() const#

Count the caller’s owned entries ghosted by each destination rank.

The returned vector is aligned with dest(). Each value is the number of local owned entries ghosted by the corresponding destination rank.

Note

Collective

Returns:: Number of entries ghosted by each destination rank.

std::array<std::vector<int>, 2> rank_type(int split_type) const#

Return neighbours in the caller’s MPI split-type group.

Forms a communicator with MPI_Comm_split_type and restricts the caller’s destination and source ranks to the resulting group. The destination ranks ghost entries owned by the caller; the source ranks own ghosts held by the caller. For example, with MPI_COMM_TYPE_SHARED, this identifies neighbouring ranks on the same shared-memory domain.

Note

Collective on comm().

Parameters:: split_type – [in] MPI split type passed to MPI_Comm_split_type, e.g. MPI_COMM_TYPE_SHARED.
Returns:: (0) Destination ranks and (1) source ranks in the split-type group. Ranks are numbered on comm(), rather than on the split communicator.

template<class Container = std::vector<std::int32_t>> class Scatterer#

#include <Scatterer.h>

A Scatterer supports the scattering and gathering of distributed data that is associated with a common::IndexMap, using MPI.

Scatter and gather operations use MPI neighbourhood collectives.

The implementation is designed for sparse communication patterns, as is typical of patterns based on an IndexMap.

A Scatterer is stateless, i.e. it provides the required information and static data for a given parallel communication pattern but does not provide any communication caches or track the status of MPI requests. Callers of a Scatterer’s members are responsible for managing buffer and MPI request handles.

Creating, copying and destroying a Scatterer are collective, since they create, duplicate and free MPI communicators. Move construction is not collective, but move assignment is, since it frees the communicators held by the assignment target.

A forward scatter sends data associated with owned/local indices to the ranks that ghost them; a reverse scatter sends ghost data back to the owning ranks, to be accumulated into the owned data. Both use the same two-step begin/end pattern, splitting the non-blocking MPI call from completion so that unrelated work can be done while communication is in flight. A round trip for a forward scatter with block size 1, where x holds the owned data and x_ghost the ghost data:

Scatterer sc(map);
std::vector<std::int64_t> send_buffer(sc.local_indices_block().size());
{
auto& idx = sc.local_indices_block();
for (std::size_t i = 0; i < idx.size(); ++i)
    send_buffer[i] = x[idx[i]];
}
std::vector<std::int64_t> recv_buffer(sc.remote_indices_block().size());
MPI_Request request = MPI_REQUEST_NULL;
sc.scatter_fwd_begin(send_buffer.data(), recv_buffer.data(), 1, request);
// ... unrelated work can be done here while communication is
// in flight, but send_buffer/recv_buffer must not be touched ...
sc.scatter_fwd_end(request);
{
auto& idx = sc.remote_indices_block();
for (std::size_t i = 0; i < idx.size(); ++i)
    x_ghost[idx[i]] = recv_buffer[i];
}

A reverse scatter follows the same pattern with the roles of ::local_indices_block and ::remote_indices_block, and of send_buffer and recv_buffer, swapped; see ::scatter_rev_begin and ::scatter_rev_end.

Template Parameters:: Container – Container type for storing the ‘local’ and ‘remote’ indices. On CPUs this is normally std::vector<std::int32_t>. For GPUs the container should store the indices on the device, e.g. using thrust::device_vector<std::int32_t>.

Public Types

using container_type = Container #: Container type used to store local and remote indices.

Public Functions

inline explicit Scatterer(const IndexMap &map)#

Create a scatterer for data with a layout described by an IndexMap.

Note

Collective.

Parameters:: map – [in] Index map that describes the parallel layout of data.

template<class U> inline Scatterer(const Scatterer<U> &s)#

Cast-copy constructor.

Create a copy of a Scatterer, where the copy uses a different storage container for indices that are used in MPI communication. Example usage includes creating from a CPU-suitable Scatterer a GPU-suitable Scatterer that can be used with GPU-aware MPI to move data between devices. This would be typical when copying a la::Vector or la::MatrixCSR to/from a GPU. When copying a vector or matrix to/from a GPU, the underlying Scatter that manages parallel communication will usually be copied too with a different storage container.

Note

Collective. The neighbourhood communicators are duplicated, so all ranks must make the copy together.

Parameters:: s – Scatterer to copy

Scatterer(const Scatterer &scatterer) = default#: Copy constructor

Note

Collective, as for the cast-copy constructor. Move instead where the original is no longer required.

Scatterer(Scatterer &&scatterer) = default#: Move constructor

Note

Not collective, unlike the copy constructors: the communicators are taken over rather than duplicated.

~Scatterer() = default#: Destructor

Note

Collective, since the communicators are freed.

Scatterer &operator=(Scatterer &&scatterer) = default#: Move assignment

Note

Collective if this Scatterer holds communicators, since assigning to it frees them.

template<typename T> inline void scatter_fwd_begin_dtype(const T *send_buffer, T *recv_buffer, MPI_Datatype type, MPI_Request &request) const#

Start a non-blocking neighbourhood collective exchange of owned data with the ranks that ghost it.

The communication is completed by calling Scatterer::scatter_fwd_end. See ::local_indices_block for how to pack send_buffer and ::remote_indices_block for how to unpack recv_buffer.

This is a differently named function rather than an overload of ::scatter_fwd_begin because the underlying type of MPI_Datatype is implementation-defined, and is an integer type in some MPI implementations, so overloading on int and MPI_Datatype is not portably unambiguous.

Note

Collective MPI operation. Every rank in the communicator must call this function, including ranks without neighbours.

Note

The send and receive buffers must not be changed or accessed until after a call to Scatterer::scatter_fwd_end.

Note

The pointers send_buffer and recv_buffer must be pointers to the data on the target device. E.g., if the send and receive buffers are allocated on a GPU, the send_buffer and recv_buffer should be device pointers.

Parameters:

send_buffer – [in] Packed local data associated with each owned local index to be sent to processes where the data is ghosted. See Scatterer::local_indices_block for the order of the buffer and how to pack.
recv_buffer – [inout] Buffer for storing received data. See Scatterer::remote_indices_block for the order of the buffer and how to unpack.
type – [in] MPI datatype for the data associated with one index, e.g. dolfinx::MPI::Datatype<T>(bs).type(). Buffer counts and displacements are in units of type, and the same type must be used on all ranks. MPI keeps a datatype alive until communication using it has completed, so type may be freed as soon as this function returns.
request – [out] Handle for tracking the status of the non-blocking communication. Any value passed in is overwritten, and MPI_REQUEST_NULL is returned when this rank has nothing to communicate. The same handle must be passed to Scatterer::scatter_fwd_end to complete the communication.

template<typename T> inline void scatter_fwd_begin(const T *send_buffer, T *recv_buffer, int bs, MPI_Request &request) const#

Start a non-blocking neighbourhood collective exchange of owned data with the ranks that ghost it.

As ::scatter_fwd_begin_dtype, but with the MPI datatype built from a block size.

Parameters:

send_buffer – [in] Packed local data associated with each owned local index to be sent to processes where the data is ghosted. See Scatterer::local_indices_block for the order of the buffer and how to pack.
recv_buffer – [inout] Buffer for storing received data. See Scatterer::remote_indices_block for the order of the buffer and how to unpack.
bs – [in] Number of values per index map index (the block size). The buffers hold bs values for each index in ::local_indices_block and ::remote_indices_block respectively.
request – [out] Handle for tracking the status of the non-blocking communication. Any value passed in is overwritten, and MPI_REQUEST_NULL is returned when this rank has nothing to communicate. The same handle must be passed to Scatterer::scatter_fwd_end to complete the communication.

inline void scatter_fwd_end(MPI_Request &request) const#

Complete a non-blocking MPI neighbourhood collective send.

This function completes the communication started by ::scatter_fwd_begin or ::scatter_fwd_begin_dtype.

Note

Local completion of the caller’s own request, not itself collective. Every rank that called ::scatter_fwd_begin must still call this before reusing the buffers.

Parameters:: request – [inout] Handle returned by the matching begin call. Set to MPI_REQUEST_NULL once the communication has completed.

template<typename T> inline void scatter_rev_begin_dtype(const T *send_buffer, T *recv_buffer, MPI_Datatype type, MPI_Request &request) const#

Start a non-blocking neighbourhood collective exchange of ghost data with the owning ranks.

The communication is completed by calling Scatterer::scatter_rev_end. See ::remote_indices_block for how to pack send_buffer and ::local_indices_block for how to unpack recv_buffer.

This is a differently named function rather than an overload of ::scatter_rev_begin because the underlying type of MPI_Datatype is implementation-defined, and is an integer type in some MPI implementations, so overloading on int and MPI_Datatype is not portably unambiguous.

Note

Collective MPI operation. Every rank in the communicator must call this function, including ranks without neighbours.

Note

The send and receive buffers must not be changed or accessed until after a call to Scatterer::scatter_rev_end.

Note

The pointers send_buffer and recv_buffer must be pointers to the data on the target device. E.g., if the send and receive buffers are allocated on a GPU, the send_buffer and recv_buffer should be device pointers.

Parameters:

send_buffer – [in] Data associated with each ghost index. This data is sent to the process that owns the index. See Scatterer::remote_indices_block for the order of the buffer and how to pack.
recv_buffer – [inout] Buffer for storing received data. See Scatterer::local_indices_block for the order of the buffer and how to unpack.
type – [in] MPI datatype for the data associated with one index, e.g. dolfinx::MPI::Datatype<T>(bs).type(). Buffer counts and displacements are in units of type, and the same type must be used on all ranks. MPI keeps a datatype alive until communication using it has completed, so type may be freed as soon as this function returns.
request – [out] Handle for tracking the status of the non-blocking communication. Any value passed in is overwritten, and MPI_REQUEST_NULL is returned when this rank has nothing to communicate. The same handle must be passed to Scatterer::scatter_rev_end to complete the communication.

template<typename T> inline void scatter_rev_begin(const T *send_buffer, T *recv_buffer, int bs, MPI_Request &request) const#

Start a non-blocking neighbourhood collective exchange of ghost data with the owning ranks.

As ::scatter_rev_begin_dtype, but with the MPI datatype built from a block size.

Parameters:

send_buffer – [in] Data associated with each ghost index. This data is sent to the process that owns the index. See Scatterer::remote_indices_block for the order of the buffer and how to pack.
recv_buffer – [inout] Buffer for storing received data. See Scatterer::local_indices_block for the order of the buffer and how to unpack.
bs – [in] Number of values per index map index (the block size). The buffers hold bs values for each index in ::remote_indices_block and ::local_indices_block respectively.
request – [out] Handle for tracking the status of the non-blocking communication. Any value passed in is overwritten, and MPI_REQUEST_NULL is returned when this rank has nothing to communicate. The same handle must be passed to Scatterer::scatter_rev_end to complete the communication.

inline void scatter_rev_end(MPI_Request &request) const#

Complete a non-blocking MPI neighbourhood collective send.

This function completes the communication started by ::scatter_rev_begin or ::scatter_rev_begin_dtype.

Note

Local completion of the caller’s own request, not itself collective. Every rank that called ::scatter_rev_begin must still call this before reusing the buffers.

Parameters:: request – [inout] Handle returned by the matching begin call. Set to MPI_REQUEST_NULL once the communication has completed.

inline const container_type &local_indices_block() const noexcept#

Array of indices for packing/unpacking owned data to/from a send/receive buffer.

For a forward scatter, the indices are used to copy required entries in the owned part of the data array into the appropriate position in a send buffer. For a reverse scatter, indices are used for assigning (accumulating) the receive buffer values into the correct position in the owned part of the data array.

The indices are in blocks, so for a block size bs a buffer holds bs values per index and must be bs * local_indices_block().size() long.

For a forward scatter, if x is the owned part of an array and send_buffer is the send buffer, send_buffer is packed such that:

auto& idx = scatterer.local_indices_block()
std::vector<T> send_buffer(bs * idx.size())
for (std::size_t i = 0; i < idx.size(); ++i)
    for (int j = 0; j < bs; ++j)
        send_buffer[i * bs + j] = x[idx[i] * bs + j];

For a reverse scatter, if recv_buffer is the received buffer, then x is updated by

auto& idx = scatterer.local_indices_block()
std::vector<T> recv_buffer(bs * idx.size())
for (std::size_t i = 0; i < idx.size(); ++i)
    for (int j = 0; j < bs; ++j)
        x[idx[i] * bs + j]
            = op(recv_buffer[i * bs + j], x[idx[i] * bs + j]);

where op is a binary operation, e.g. x[...] = buffer[...] or x[...] += buffer[...].

Returns:: Indices container.

inline const container_type &remote_indices_block() const noexcept#

Array of indices for packing/unpacking ghost data to/from a send/receive buffer.

For a forward scatter, the indices are used to unpack received data into ghost entries. For a reverse scatter, indices are used to pack ghost entries into the send buffer.

The indices are in blocks, so for a block size bs a buffer holds bs values per index and must be bs * remote_indices_block().size() long.

For a forward scatter, if xg is the ghost part of the data array and recv_buffer is the receive buffer, xg is updated as

auto& idx = scatterer.remote_indices_block()
std::vector<T> recv_buffer(bs * idx.size())
for (std::size_t i = 0; i < idx.size(); ++i)
    for (int j = 0; j < bs; ++j)
        xg[idx[i] * bs + j] = recv_buffer[i * bs + j];

For a reverse scatter, if send_buffer is the send buffer, then send_buffer is packed such that:

auto& idx = scatterer.remote_indices_block()
std::vector<T> send_buffer(bs * idx.size())
for (std::size_t i = 0; i < idx.size(); ++i)
    for (int j = 0; j < bs; ++j)
        send_buffer[i * bs + j] = xg[idx[i] * bs + j];

Returns:: Block indices container.

class TimeLogger#

#include <TimeLogger.h>

Time logger maintaining data collected by Timer, if registered.

Note

This is a monotstate, i.e. the data members are static and thus timings are aggregated into a single map.

Public Functions

void register_timing(std::string_view task, std::chrono::duration<double, std::ratio<1>> wall)#: Register timing (for later summary).

Table timing_table() const#: Return a summary of timings and tasks in a Table.

void list_timings(MPI_Comm comm, Table::Reduction reduction) const#

List a summary of timings and tasks. Reduction type is printed.

Parameters:

comm – MPI Communicator
reduction – Reduction type (min, max or average)

std::pair<int, std::chrono::duration<double, std::ratio<1>>> timing(std::string_view task) const#

Return timing.

Parameters:: task – [in] The task name to retrieve the timing for
Returns:: Values (count, total wall time) for given task.

std::map<std::string, std::pair<int, std::chrono::duration<double, std::ratio<1>>>, std::less<>> timings() const#

Logged elapsed times.

Returns:: Elapsed [task id: (count, total wall time)].

Public Static Functions

static TimeLogger &instance()#

Singleton access.

Returns:: Unique time logger object.

template<typename T = std::chrono::high_resolution_clock> class Timer#

#include <Timer.h>

Timer for measuring and logging elapsed time durations.

The basic usage is

Timer timer("Assembling over cells");

The timer is started at construction and timing ends when the timer is destroyed (goes out-of-scope). The timer can be started (reset) and stopped explicitly by

timer.start();
 /* .... */
timer.stop();

A summary of registered elapsed times can be printed by calling:

list_timings();

Registered elapsed times are logged when (1) the timer goes out-of-scope or (2) Timer::flush() is called.

Public Functions

inline Timer(std::optional<std::string> task = std::nullopt)#

Create and start timer.

Elapsed time is optionally registered in the logger when the Timer destructor is called.

Parameters:: task – [in] Name used to registered the elapsed time in the logger. If no name is set, the elapsed time is not registered in the logger.

inline Timer(Timer &&timer) noexcept#: Move constructor.

Note

Resets the moved-from timer explicitly. A moved-from std::optional stays engaged, so a defaulted move would leave the source registering an empty task name.

inline ~Timer()#: If timer is still running, it is stopped. Elapsed time is registered in the logger.

inline Timer &operator=(Timer &&timer) noexcept#: Move assignment.

Note

Any elapsed time not yet registered by the target is discarded. See the move constructor for why the source is reset.

inline void start()#: Reset elapsed time and (re-)start timer.

template<typename Period = std::ratio<1>> inline std::chrono::duration<double, Period> elapsed() const#

Elapsed time since time has been started.

Default duration unit is seconds.

Returns:: Elapsed time duration.

template<typename Period = std::ratio<1>> inline std::chrono::duration<double, Period> stop()#

Stop timer and return elapsed time.

Default duration unit is seconds.

Returns:: Elapsed time duration.

inline void resume()#

Resume a stopped timer.

Does nothing if timer has not been stopped.

inline void flush()#

Flush timer duration to the logger.

An instance of a timer can be flushed to the logger only once. Subsequent calls will have no effect and will not trigger any logging.

Pre:: Timer must have been stopped before flushing.

namespace petsc#

PETSc error handling helpers shared across DOLFINx’s PETSc wrappers.

Functions

void error(PetscErrorCode error_code, std::string_view petsc_function, std::source_location loc = std::source_location::current())#

Print error message for a PETSc call that returned an error and throw a std::runtime_error.

Parameters:

error_code – [in] PETSc error code
petsc_function – [in] Name of the PETSc function that returned error_code
loc – [in] Call site of the failed PETSc call (captured automatically; do not pass explicitly)

inline void check(PetscErrorCode ierr, std::string_view petsc_function, std::source_location loc = std::source_location::current())#

Throw a std::runtime_error via error() if ierr indicates a PETSc call failed.

Parameters:

ierr – [in] PETSc error code returned by petsc_function
petsc_function – [in] Name of the PETSc function that returned ierr
loc – [in] Call site of the failed PETSc call (captured automatically; do not pass explicitly)

void set_option(std::string option)#

Set a PETSc option in the PETSc options/parameter database. The option must not be prefixed by ‘-’, e.g.

common::petsc::set_option("mat_mumps_icntl_14", 40);

template<typename T> void set_option(std::string option, const T &value)#

Set a PETSc option that takes a value in the PETSc options/parameter database. The option must not be prefixed by ‘-’, e.g.

common::petsc::set_option("mat_mumps_icntl_14", 40);

void clear_option(std::string option)#: Clear a PETSc option from the PETSc options/parameter database.

void clear_options()#: Clear the PETSc global options/parameter database.

Common (dolfinx::common)#

Common (`dolfinx::common`)#