Common (`dolfinx::common`)

namespace common

Miscellaneous classes, functions and types.

Generic tools.

This namespace provides utility type functions for managing subsystems, convenience classes and library-wide typedefs.

Enums

enum class IndexMapOrder : bool

Enum to control preservation of ghost index ordering in sub-IndexMaps.

Values:

enumerator preserve: Preserve the ordering of ghost indices.

enumerator any: Allow arbitrary ordering of ghost indices in sub-maps.

Functions

std::vector<int32_t> compute_owned_indices(std::span<const std::int32_t> indices, const IndexMap &map)

Given a sorted list of indices (local indexing, owned or ghost) and an index map, this function returns the indices owned by this process, including indices that might have been in the list of indices on another processes.

Parameters:

indices – [in] List of indices.
map – [in] The index map.

Returns:

Indices owned by the calling process.

std::tuple<std::int64_t, std::vector<std::int32_t>, std::vector<std::vector<std::int64_t>>, std::vector<std::vector<int>>> stack_index_maps(const std::vector<std::pair<std::reference_wrapper<const IndexMap>, int>> &maps)

Compute layout data and ghost indices for a stacked (concatenated) index map, i.e. ‘splice’ multiple maps into one.

The input maps are concatenated, with indices in maps and owned by the caller remaining owned by the caller. Ghost data is stored at the end of the local range as normal, with the ghosts in blocks in the order of the index maps in maps.

Note

Index maps with a block size are unrolled in the data for the concatenated index map.

Note

Communication is required to compute the new ghost indices.

Parameters:: maps – [in] List of (index map, block size) pairs
Returns:: The (0) global offset of a concatenated map for the calling rank, (1) local offset for the owned indices of each submap in the concatenated map, (2) new indices for the ghosts for each submap, and (3) owner rank of each ghost entry for each submap.

std::pair<IndexMap, std::vector<std::int32_t>> create_sub_index_map(const IndexMap &imap, std::span<const std::int32_t> indices, IndexMapOrder order = IndexMapOrder::any, bool allow_owner_change = false)

Create a new index map from a subset of indices in an existing index map.

Parameters:

imap – [in] Parent map to create a new sub-map from.
indices – [in] Local indices in imap (owned and ghost) to include in the new index map.
order – [in] Control the order in which ghost indices appear in the new map.
allow_owner_change – [in] If true, indices that are not included in indices by their owning process can be included in indices by processes that ghost the indices to be included in the new submap. These indices will be owned by one of the sharing processes in the submap. If false, and exception is raised if an index is included by a sharing process and not by the owning process.

Returns:

The (i) new index map and (ii) a map from local indices in the submap to local indices in the original (this) map.

Pre:

indices must be sorted and must not contain duplicates.

template<typename U, typename V> std::pair<std::vector<typename U::value_type>, std::vector<typename V::value_type>> sort_unique(const U &indices, const V &values)

Sort two arrays based on the values in array indices.

Any duplicate indices and the corresponding value are removed. In the case of duplicates, the entry with the smallest value is retained.

Parameters:

indices – [in] Array of indices.
values – [in] Array of values.

Returns:

Sorted (indices, values), with sorting based on indices.

template<class T> std::size_t hash_local(const T &x)

Compute a hash of a given object.

The hash is computed using Boost container hash (https://www.boost.org/doc/libs/release/libs/container_hash/).

Parameters:: x – [in] The object to compute a hash of.
Returns:: The hash values.

template<class T> std::size_t hash_global(MPI_Comm comm, const T &x)

Compute a hash for a distributed (MPI) object.

A hash is computed on each process for the local part of the object. Then, a hash of the std::vector containing each local hash key in rank order is returned.

Note

Collective

Parameters:

comm – [in] The communicator on which to compute the hash.
x – [in] The object to compute a hash of.

Returns:

The hash values.

std::string comm_to_json(const graph::AdjacencyList<std::tuple<int, std::size_t, std::int8_t>, std::pair<std::int32_t, std::int32_t>> &g)

Build communication graph data as a JSON string.

The data string can be decoded (loaded) to create a Python object from which a NetworkX graph can be constructed.

See ::comm_graph for a description of the data.

Parameters:: g – [in] Communication graph.
Returns:: JSON string representing the communication graph. Edge data is data volume (weight) and local/remote memory indicator (local==1 is an edge to an shared memory process/rank, other wise the target node is a remote memory rank).

class IndexMap

#include <IndexMap.h>

This class represents the distribution index arrays across processes. An index array is a contiguous collection of N+1 indices [0, 1, . . ., N] that are distributed across M processes. On a given process, the IndexMap stores a portion of the index set using local indices [0, 1, . . . , n], and a map from the local indices to a unique global index.

Public Functions

IndexMap(MPI_Comm comm, std::int32_t local_size)

Create an non-overlapping index map.

Note

Collective

Parameters:

comm – [in] MPI communicator that the index map is distributed across.
local_size – [in] Local size of the index map, i.e. the number of owned entries.

IndexMap(MPI_Comm comm, std::int32_t local_size, std::span<const std::int64_t> ghosts, std::span<const int> owners, int tag = static_cast<int>(dolfinx::MPI::tag::consensus_nbx))

Create an overlapping (ghosted) index map.

This constructor uses a ‘consensus’ algorithm to determine the ranks that ghost indices that are owned by the caller. This requires non-trivial MPI communication. If the ranks that ghost indices owned by the caller are known, it more efficient to use the constructor that takes these ranks as an argument.

Note

Collective

Note

A tag can sometimes be required when there are a series of calls to this constructor, or other functions that call the consensus algorithm, that are close together. In cases where this constructor is called a second time on rank and another rank has not completed its first consensus algorithm call, communications can be corrupted if each collective call of this constructor does not have its own tag value. Each collective call to this constructor must use the same tag value. An alternative to passing a tag is to have an implicit or explicit MPI barrier before and after the call to this constructor.

Parameters:

comm – [in] MPI communicator that the index map is distributed across.
local_size – [in] Local size of the index map, i.e. the number of owned entries
ghosts – [in] The global indices of ghost entries
owners – [in] Owner rank (on comm) of each entry in ghosts
tag – [in] Tag used in non-blocking MPI calls in the consensus algorithm.

IndexMap(MPI_Comm comm, std::int32_t local_size, const std::array<std::vector<int>, 2> &src_dest, std::span<const std::int64_t> ghosts, std::span<const int> owners)

Create an overlapping (ghosted) index map.

This constructor is optimised for the case where the ‘source’ (ranks that own indices ghosted by the caller) and ‘destination’ ranks (ranks that ghost indices owned by the caller) are already available. It allows the complex computation of the destination ranks from owners.

Note

Collective

Parameters:

comm – [in] MPI communicator that the index map is distributed across.
local_size – [in] Local size of the index map, i.e. the number
src_dest – [in] Lists of [0] src and [1] dest ranks. The list in each must be sorted and not contain duplicates. src ranks are owners of the indices in ghosts. dest ranks are the rank that ghost indices owned by the caller.
ghosts – [in] The global indices of ghost entries
owners – [in] Owner rank (on comm) of each entry in ghosts

IndexMap(IndexMap &&map) = default: Move constructor.

~IndexMap() = default: Destructor.

IndexMap &operator=(IndexMap &&map) = default: Move assignment.

std::array<std::int64_t, 2> local_range() const noexcept: Range of indices (global) owned by this process.

std::int32_t num_ghosts() const noexcept: Number of ghost indices on this process.

std::int32_t size_local() const noexcept: Number of indices owned by this process.

std::int64_t size_global() const noexcept: Number indices across communicator.

std::span<const std::int64_t> ghosts() const noexcept: Local-to-global map for ghosts (local indexing beyond end of local range)

MPI_Comm comm() const

Return the MPI communicator that the map is defined on.

Returns:: Communicator

void local_to_global(std::span<const std::int32_t> local, std::span<std::int64_t> global) const

Compute global indices for array of local indices.

Parameters:

local – [in] Local indices
global – [out] The global indices

void global_to_local(std::span<const std::int64_t> global, std::span<std::int32_t> local) const

Compute local indices for array of global indices.

Parameters:

global – [in] Global indices
local – [out] The local of the corresponding global index in ‘global’. Returns -1 if the local index does not exist on this process.

std::vector<std::int64_t> global_indices() const

Build list of indices with global indexing.

Returns:: The global index for all local indices (0, 1, 2, ...) on this process, including ghosts

inline std::span<const int> owners() const

The ranks that own each ghost index.

Returns:: List of ghost owners. The owning rank of the ith ghost index is owners()[i].

graph::AdjacencyList<int> index_to_dest_ranks(int tag = static_cast<int>(dolfinx::MPI::tag::consensus_nbx)) const

Compute map from each local (owned) index to the set of ranks that have the index as a ghost.

Todo:: Aim to remove this function?

Note

Collective

Note

See IndexMap(MPI_Comm,std::int32_t,std::span<conststd::int64_t>,std::span<const int>,int) for an explanation of when tag is required.

Parameters:: tag – [in] Tag to pass to MPI calls.
Returns:: Shared indices.

std::vector<std::int32_t> shared_indices() const

Build a list of owned indices that are ghosted by another rank.

Returns:: The local index of owned indices that are ghosts on other rank(s). The indices are unique and sorted.

std::span<const int> src() const noexcept

Ordered set of MPI ranks that own caller’s ghost indices.

Typically used when creating neighbourhood communicators.

Returns:: MPI ranks than own ghost indices. The ranks are unique and sorted.

std::span<const int> dest() const noexcept

Ordered set of MPI ranks that ghost indices owned by caller.

Typically used when creating neighbourhood communicators.

Returns:: MPI ranks than own ghost indices. The ranks are unique and sorted.

std::vector<std::int32_t> weights_src() const

Compute the number of ghost indices owned by each rank in IndexMap::src.

This is a measure of the amount of data:

Sent from this rank to other ranks when performing a reverse (owner <- ghost) scatter.
Received by this rank from other ranks when performing a forward (owner -> ghost) scatter.

Returns:: A weight vector, where weight[i] the the number of ghost indices owned by rank IndexMap::src()[i].

std::vector<std::int32_t> weights_dest() const

Compute the number of ghost indices owned by each rank in IndexMap::dest.

This is a measure of the amount of data:

Sent from this rank to other ranks when performing a forward (owner -> ghost) scatter.
Received by this rank from other ranks when performing a reverse forward (owner <- ghost) scatter.

Returns:: A weight vector, where weight[i] the the number of ghost indices owned by rank IndexMap::dest()[i].

std::array<std::vector<int>, 2> rank_type(int split_type) const

Destination and source ranks by type, e.g, ranks that are destination/source ranks for the caller and are in a common shared memory region.

This function is used to group destination and source ranks by ‘type’. The type is defined by the MPI split_type. Split types include ranks from a common shared memory region (MPI_COMM_TYPE_SHARED) or a common NUMA region. Splits types are listed at https://docs.open-mpi.org/en/main/man-openmpi/man3/MPI_Comm_split_type.3.html#split-types.

Note

Collective operation on comm().

Parameters:: split_type – [in] MPI split type, as used in the function MPI_Comm_split_type. See https://docs.open-mpi.org/en/main/man-openmpi/man3/MPI_Comm_split_type.3.html#split-types.
Returns:: (0) Intersection of ranks in split_type and in dest(), and (1) intersection of ranks in split_type and in src(). Returned ranks are on the comm() communicator.

template<class Container = std::vector<std::int32_t>> class Scatterer

#include <Scatterer.h>

A Scatterer supports the scattering and gathering of distributed data that is associated with a common::IndexMap, using MPI.

Scatter and gather operations can use:

MPI neighbourhood collectives (recommended), or
Non-blocking point-to-point communication modes.

The implementation is designed for sparse communication patterns, as is typical of patterns based on an IndexMap.

Template Parameters:: Container – Container type for storing the ‘local’ and ‘remote’ indices. On CPUs this is normally std::vector<std::int32_t>. For GPUs the container should store the indices on the device, e.g. using thrust::device_vector<std::int32_t>.

Public Types

using container_type = Container : Container type used to store local and remote indices.

Public Functions

inline Scatterer(const IndexMap &map, int bs)

Create a scatterer for data with a layout described by an IndexMap, and with a block size.

Parameters:

map – [in] Index map that describes the parallel layout of data.
bs – [in] Number of values associated with each map index (block size).

template<typename T> inline void scatter_fwd_begin(const T *send_buffer, T *recv_buffer, MPI_Request &request) const

Start a non-blocking send of owned data to ranks that ghost the data using MPI neighbourhood collective communication (recommended).

The communication is completed by calling Scatterer::scatter_end. See local_indices for instructions on packing send_buffer and remote_indices for instructions on unpacking recv_buffer.

Note

The send and receive buffers must not to be changed or accessed until after a call to Scatterer::scatter_end.

Note

The pointers send_buffer and recv_buffer must be pointers to the data on the target device. E.g., if the send and receive buffers are allocated on a GPU, the send_buffer and recv_buffer should be device pointers.

Parameters:

send_buffer – [in] Packed local data associated with each owned local index to be sent to processes where the data is ghosted. See Scatterer::local_indices for the order of the buffer and how to pack.
recv_buffer – [inout] Buffer for storing received data. See Scatterer::remote_indices for the order of the buffer and how to unpack.
request – [in] MPI request handle for tracking the status of the non-blocking communication. The same request handle should be passed to Scatterer::scatter_end to complete the communication.

template<typename T> inline void scatter_fwd_begin(const T *send_buffer, T *recv_buffer, std::span<MPI_Request> requests) const

Start a non-blocking send of owned data to ranks that ghost the data using point-to-point MPI communication.

See scatter_fwd_begin for a detailed explanation of usage, including on the send and receive buffer packing and unpacking

Note

Use of the neighbourhood version of scatter_fwd_begin is recommended over this version.

Parameters:

send_buffer – [in] Send buffer.
recv_buffer – [inout] Receive buffer.
requests – [in] List of MPI request handles. The length of the list must be num_p2p_requests()

template<typename T> inline void scatter_rev_begin(const T *send_buffer, T *recv_buffer, MPI_Request &request) const

Start a non-blocking send of ghost data to ranks that own the data using MPI neighbourhood collective communication (recommended).

The communication is completed by calling Scatterer::scatter_end. See remote_indices for instructions on packing send_buffer and local_indices for instructions on unpacking recv_buffer.

Note

The send and receive buffers must not to be changed or accessed until after a call to Scatterer::scatter_end.

Note

The pointers send_buffer and recv_buffer must be pointers to the data on the target device. E.g., if the send and receive buffers are allocated on a GPU, the send_buffer and recv_buffer should be device pointers.

Parameters:

send_buffer – [in] Data associated with each ghost index. This data is sent to process that owns the index. See Scatterer::remote_indices for the order of the buffer and how to pack.
recv_buffer – [inout] Buffer for storing received data. See Scatterer::local_indices for the order of the buffer and how to unpack.
request – [in] MPI request handle for tracking the status of the non-blocking communication. The same request handle should be passed to Scatterer::scatter_end to complete the communication.

template<typename T> inline void scatter_rev_begin(const T *send_buffer, T *recv_buffer, std::span<MPI_Request> requests) const

Start a non-blocking send of ghost data to ranks that own the data using point-to-point MPI communication.

See scatter_rev_begin for a detailed explanation of usage, including on the send and receive buffer packing and unpacking

Note

Use of the neighbourhood version of scatter_rev_begin is recommended over this version

Parameters:

send_buffer – [in] Send buffer.
recv_buffer – [inout] Receive buffer.
requests – [in] List of MPI request handles. The length of the list must be num_p2p_requests()

inline void scatter_end(std::span<MPI_Request> requests) const

Complete non-blocking MPI point-to-point sends.

This function completes the communication started by scatter_fwd_begin or scatter_rev_begin.

Parameters:: requests – [in] MPI request handles for tracking the status of sends.

inline void scatter_end(MPI_Request &request) const

Complete a non-blocking MPI neighbourhood collective send.

This function completes the communication started by scatter_fwd_begin or scatter_rev_begin.

Parameters:: request – [in] MPI request handle for tracking the status of the send.

inline const container_type &local_indices() const noexcept

Array of indices for packing/unpacking owned data to/from a send/receive buffer.

For a forward scatter, the indices are used to copy required entries in the owned part of the data array into the appropriate position in a send buffer. For a reverse scatter, indices are used for assigning (accumulating) the receive buffer values into correct position in the owned part of the data array.

For a forward scatter, if x is the owned part of an array and send_buffer is the send buffer, send_buffer is packed such that:

auto& idx = scatterer.local_indices()
std::vector<T> send_buffer(idx.size())
for (std::size_t i = 0; i < idx.size(); ++i)
    send_buffer[i] = x[idx[i]];

For a reverse scatter, if recv_buffer is the received buffer, then x is updated by

auto& idx = scatterer.local_indices()
std::vector<T> recv_buffer(idx.size())
for (std::size_t i = 0; i < idx.size(); ++i)
    x[idx[i]] = op(recv_buffer[i], x[idx[i]]);

where op is a binary operation, e.g. x[idx[i]] = buffer[i] or x[idx[i]] += buffer[i].

Returns:: Indices container.

inline const container_type &remote_indices() const noexcept

Array of indices for packing/unpacking ghost data to/from a send/receive buffer.

For a forward scatter, the indices are to copy required entries in the owned array into the appropriate position in a send buffer. For a reverse scatter, indices are used for assigning (accumulating) the receive buffer values to correct position in the owned array.

For a forward scatter, if xg is the ghost part of the data array and recv_buffer is the receive buffer, xg is updated that

auto& idx = scatterer.remote_indices()
std::vector<T> recv_buffer(idx.size())
for (std::size_t i = 0; i < idx.size(); ++i)
    xg[idx[i]] = recv_buffer[i];

For a reverse scatter, if send_buffer is the send buffer, then send_buffer is packaged such that:

auto& idx = scatterer.local_indices()
std::vector<T> send_buffer(idx.size())
for (std::size_t i = 0; i < idx.size(); ++i)
    send_buffer[i] = xg[idx[i]];

Returns:: Indices container.

inline std::size_t num_p2p_requests()

Number of required MPI_Requests for point-to-point communication.

Returns:: Numer of required MPI request handles.

class TimeLogger

#include <TimeLogger.h>

Time logger maintaining data collected by Timer, if registered.

Note

This is a monotstate, i.e. the data members are static and thus timings are aggregated into a single map.

Public Functions

void register_timing(const std::string &task, std::chrono::duration<double, std::ratio<1>> wall): Register timing (for later summary)

Table timing_table() const: Return a summary of timings and tasks in a Table.

void list_timings(MPI_Comm comm, Table::Reduction reduction) const

List a summary of timings and tasks. Reduction type is printed.

Parameters:

comm – MPI Communicator
reduction – Reduction type (min, max or average)

std::pair<int, std::chrono::duration<double, std::ratio<1>>> timing(const std::string &task) const

Return timing.

Parameters:: task – [in] The task name to retrieve the timing for
Returns:: Values (count, total wall time) for given task.

std::map<std::string, std::pair<int, std::chrono::duration<double, std::ratio<1>>>> timings() const

Logged elapsed times.

Returns:: Elapsed [task id: (count, total wall time)].

Public Static Functions

static TimeLogger &instance()

Singleton access.

Returns:: Unique time logger object.

template<typename T = std::chrono::high_resolution_clock> class Timer

#include <Timer.h>

Timer for measuring and logging elapsed time durations.

The basic usage is

Timer timer("Assembling over cells");

The timer is started at construction and timing ends when the timer is destroyed (goes out-of-scope). The timer can be started (reset) and stopped explicitly by

timer.start();
 /* .... */
timer.stop();

A summary of registered elapsed times can be printed by calling:

list_timings();

Registered elapsed times are logged when (1) the timer goes out-of-scope or (2) Timer::flush() is called.

Public Functions

inline Timer(std::optional<std::string> task = std::nullopt)

Create and start timer.

Elapsed time is optionally registered in the logger when the Timer destructor is called.

Parameters:: task – [in] Name used to registered the elapsed time in the logger. If no name is set, the elapsed time is not registered in the logger.

inline ~Timer(): If timer is still running, it is stopped. Elapsed time is registered in the logger.

inline void start(): Reset elapsed time and (re-)start timer.

template<typename Period = std::ratio<1>> inline std::chrono::duration<double, Period> elapsed() const

Elapsed time since time has been started.

Default duration unit is seconds.

Returns:: Elapsed time duration.

template<typename Period = std::ratio<1>> inline std::chrono::duration<double, Period> stop()

Stop timer and return elapsed time.

Default duration unit is seconds.

Returns:: Elapsed time duration.

inline void resume()

Resume a stopped timer.

Does nothing if timer has not been stopped.

inline void flush()

Flush timer duration to the logger.

An instance of a timer can be flushed to the logger only once. Subsequent calls will have no effect and will not trigger any logging.

Pre:: Timer must have been stopped before flushing.

Common (dolfinx::common)

Common (`dolfinx::common`)