Common (dolfinx::common)

namespace common

Miscellaneous classes, functions and types.

Generic tools.

This namespace provides utility type functions for managing subsystems, convenience classes and library-wide typedefs.

Enums

enum class IndexMapOrder : bool

Enum to control preservation of ghost index ordering in sub-IndexMaps.

Values:

enumerator preserve

Preserve the ordering of ghost indices.

enumerator any

Allow arbitrary ordering of ghost indices in sub-maps.

Functions

std::vector<int32_t> compute_owned_indices(std::span<const std::int32_t> indices, const IndexMap &map)

Given a sorted vector of indices (local numbering, owned or ghost) and an index map, this function returns the indices owned by this process, including indices that might have been in the list of indices on another processes.

Parameters:

  • indices – [in] List of indices

  • map – [in] The index map

Returns:

Indices owned by the calling process

std::tuple<std::int64_t, std::vector<std::int32_t>, std::vector<std::vector<std::int64_t>>, std::vector<std::vector<int>>> stack_index_maps(const std::vector<std::pair<std::reference_wrapper<const IndexMap>, int>> &maps)

Compute layout data and ghost indices for a stacked (concatenated) index map, i.e. ‘splice’ multiple maps into one.

The input maps are concatenated, with indices in maps and owned by the caller remaining owned by the caller. Ghost data is stored at the end of the local range as normal, with the ghosts in blocks in the order of the index maps in maps.

Note

Index maps with a block size are unrolled in the data for the concatenated index map.

Note

Communication is required to compute the new ghost indices.

Parameters:

maps – [in] List of (index map, block size) pairs

Returns:

The (0) global offset of a concatenated map for the calling rank, (1) local offset for the owned indices of each submap in the concatenated map, (2) new indices for the ghosts for each submap, and (3) owner rank of each ghost entry for each submap.

std::pair<IndexMap, std::vector<std::int32_t>> create_sub_index_map(const IndexMap &imap, std::span<const std::int32_t> indices, IndexMapOrder order = IndexMapOrder::any, bool allow_owner_change = false)

Create a new index map from a subset of indices in an existing index map.

Parameters:

  • imap – [in] Parent map to create a new sub-map from.

  • indices – [in] Local indices in imap (owned and ghost) to include in the new index map.

  • order – [in] Control the order in which ghost indices appear in the new map.

  • allow_owner_change – [in] If true, indices that are not included in indices by their owning process can be included in indices by processes that ghost the indices to be included in the new submap. These indices will be owned by one of the sharing processes in the submap. If false, and exception is raised if an index is included by a sharing process and not by the owning process.

Returns:

The (i) new index map and (ii) a map from local indices in the submap to local indices in the original (this) map.

Pre:

indices must be sorted and must not contain duplicates.

template<typename U, typename V>
std::pair<std::vector<typename U::value_type>, std::vector<typename V::value_type>> sort_unique(const U &indices, const V &values)

Sort two arrays based on the values in array indices.

Any duplicate indices and the corresponding value are removed. In the case of duplicates, the entry with the smallest value is retained.

Parameters:

  • indices – [in] Array of indices.

  • values – [in] Array of values.

Returns:

Sorted (indices, values), with sorting based on indices.

template<class T>
std::size_t hash_local(const T &x)

Compute a hash of a given object.

The hash is computed using Boost container hash (https://www.boost.org/doc/libs/release/libs/container_hash/).

Parameters:

x – [in] The object to compute a hash of.

Returns:

The hash values.

template<class T>
std::size_t hash_global(MPI_Comm comm, const T &x)

Compute a hash for a distributed (MPI) object.

A hash is computed on each process for the local part of the object. Then, a hash of the std::vector containing each local hash key in rank order is returned.

Note

Collective

Parameters:

  • comm – [in] The communicator on which to compute the hash.

  • x – [in] The object to compute a hash of.

Returns:

The hash values.

class IndexMap
#include <IndexMap.h>

This class represents the distribution index arrays across processes. An index array is a contiguous collection of N+1 indices [0, 1, . . ., N] that are distributed across M processes. On a given process, the IndexMap stores a portion of the index set using local indices [0, 1, . . . , n], and a map from the local indices to a unique global index.

Public Functions

IndexMap(MPI_Comm comm, std::int32_t local_size)

Create an non-overlapping index map.

Note

Collective

Parameters:

  • comm – [in] MPI communicator that the index map is distributed across.

  • local_size – [in] Local size of the index map, i.e. the number of owned entries.

IndexMap(MPI_Comm comm, std::int32_t local_size, std::span<const std::int64_t> ghosts, std::span<const int> owners, int tag = static_cast<int>(dolfinx::MPI::tag::consensus_nbx))

Create an overlapping (ghosted) index map.

This constructor uses a ‘consensus’ algorithm to determine the ranks that ghost indices that are owned by the caller. This requires non-trivial MPI communication. If the ranks that ghost indices owned by the caller are known, it more efficient to use the constructor that takes these ranks as an argument.

Note

Collective

Note

A tag can sometimes be required when there are a series of calls to this constructor, or other functions that call the consensus algorithm, that are close together. In cases where this constructor is called a second time on rank and another rank has not completed its first consensus algorithm call, communications can be corrupted if each collective call of this constructor does not have its own tag value. Each collective call to this constructor must use the same tag value. An alternative to passing a tag is to have an implicit or explicit MPI barrier before and after the call to this constructor.

Parameters:

  • comm – [in] MPI communicator that the index map is distributed across.

  • local_size – [in] Local size of the index map, i.e. the number of owned entries

  • ghosts – [in] The global indices of ghost entries

  • owners – [in] Owner rank (on comm) of each entry in ghosts

  • tag – [in] Tag used in non-blocking MPI calls in the consensus algorithm.

IndexMap(MPI_Comm comm, std::int32_t local_size, const std::array<std::vector<int>, 2> &src_dest, std::span<const std::int64_t> ghosts, std::span<const int> owners)

Create an overlapping (ghosted) index map.

This constructor is optimised for the case where the ‘source’ (ranks that own indices ghosted by the caller) and ‘destination’ ranks (ranks that ghost indices owned by the caller) are already available. It allows the complex computation of the destination ranks from owners.

Note

Collective

Parameters:

  • comm – [in] MPI communicator that the index map is distributed across.

  • local_size – [in] Local size of the index map, i.e. the number

  • src_dest – [in] Lists of [0] src and [1] dest ranks. The list in each must be sorted and not contain duplicates. src ranks are owners of the indices in ghosts. dest ranks are the rank that ghost indices owned by the caller.

  • ghosts – [in] The global indices of ghost entries

  • owners – [in] Owner rank (on comm) of each entry in ghosts

IndexMap(IndexMap &&map) = default

Move constructor.

~IndexMap() = default

Destructor.

IndexMap &operator=(IndexMap &&map) = default

Move assignment.

std::array<std::int64_t, 2> local_range() const noexcept

Range of indices (global) owned by this process.

std::int32_t num_ghosts() const noexcept

Number of ghost indices on this process.

std::int32_t size_local() const noexcept

Number of indices owned by this process.

std::int64_t size_global() const noexcept

Number indices across communicator.

std::span<const std::int64_t> ghosts() const noexcept

Local-to-global map for ghosts (local indexing beyond end of local range)

MPI_Comm comm() const

Return the MPI communicator that the map is defined on.

Returns:

Communicator

void local_to_global(std::span<const std::int32_t> local, std::span<std::int64_t> global) const

Compute global indices for array of local indices.

Parameters:

  • local – [in] Local indices

  • global – [out] The global indices

void global_to_local(std::span<const std::int64_t> global, std::span<std::int32_t> local) const

Compute local indices for array of global indices.

Parameters:

  • global – [in] Global indices

  • local – [out] The local of the corresponding global index in ‘global’. Returns -1 if the local index does not exist on this process.

std::vector<std::int64_t> global_indices() const

Build list of indices with global indexing.

Returns:

The global index for all local indices (0, 1, 2, ...) on this process, including ghosts

inline std::span<const int> owners() const

The ranks that own each ghost index.

Returns:

List of ghost owners. The owning rank of the ith ghost index is owners()[i].

graph::AdjacencyList<int> index_to_dest_ranks(int tag = static_cast<int>(dolfinx::MPI::tag::consensus_nbx)) const

Compute map from each local (owned) index to the set of ranks that have the index as a ghost.

Todo:

Aim to remove this function?

Note

Collective

Note

See IndexMap(MPI_Comm,std::int32_t,std::span<conststd::int64_t>,std::span<const int>,int) for an explanation of when tag is required.

Parameters:

tag – [in] Tag to pass to MPI calls.

Returns:

Shared indices.

std::vector<std::int32_t> shared_indices() const

Build a list of owned indices that are ghosted by another rank.

Returns:

The local index of owned indices that are ghosts on other rank(s). The indices are unique and sorted.

std::span<const int> src() const noexcept

Ordered set of MPI ranks that own caller’s ghost indices.

Typically used when creating neighbourhood communicators.

Returns:

MPI ranks than own ghost indices. The ranks are unique and sorted.

std::span<const int> dest() const noexcept

Ordered set of MPI ranks that ghost indices owned by caller.

Typically used when creating neighbourhood communicators.

Returns:

MPI ranks than own ghost indices. The ranks are unique and sorted.

std::array<double, 2> imbalance() const

Returns the imbalance of the current IndexMap.

The imbalance is a measure of load balancing across all processes, defined as the maximum number of indices on any process divided by the average number of indices per process. This function calculates the imbalance separately for owned indices and ghost indices and returns them as a std::array<double, 2>. If the total number of owned or ghost indices is zero, the respective entry in the array is set to -1.

Note

This is a collective operation and must be called by all processes in the communicator associated with the IndexMap.

Returns:

An array containing the imbalance in owned indices (first element) and the imbalance in ghost indices (second element).

template<class Allocator = std::allocator<std::int32_t>>
class Scatterer
#include <Scatterer.h>

A Scatterer supports the MPI scattering and gathering of data that is associated with a common::IndexMap.

Scatter and gather operations use MPI neighbourhood collectives. The implementation is designed for sparse communication patterns, as it typical of patterns based on an IndexMap.

Public Types

enum class type

Types of MPI communication pattern used by the Scatterer.

Values:

enumerator neighbor
enumerator p2p
using allocator_type = Allocator

The allocator type.

Public Functions

inline Scatterer(const IndexMap &map, int bs, const Allocator &alloc = Allocator())

Create a scatterer.

Parameters:

  • map – [in] The index map that describes the parallel layout of data.

  • bs – [in] The block size of data associated with each index in map that will be scattered/gathered.

  • alloc – [in] The memory allocator for indices.

template<typename T>
inline void scatter_fwd_begin(std::span<const T> send_buffer, std::span<T> recv_buffer, std::span<MPI_Request> requests, Scatterer::type type = type::neighbor) const

Start a non-blocking send of owned data to ranks that ghost the data.

The communication is completed by calling Scatterer::scatter_fwd_end. The send and receive buffer should not be changed until after Scatterer::scatter_fwd_end has been called.

Parameters:

  • send_buffer – [in] Local data associated with each owned local index to be sent to process where the data is ghosted. It must not be changed until after a call to Scatterer::scatter_fwd_end. The order of data in the buffer is given by Scatterer::local_indices.

  • recv_buffer – A buffer used for the received data. The position of ghost entries in the buffer is given by Scatterer::remote_indices. The buffer must not be accessed or changed until after a call to Scatterer::scatter_fwd_end.

  • requests – The MPI request handle for tracking the status of the non-blocking communication

  • type – [in] The type of MPI communication pattern used by the Scatterer, either Scatterer::type::neighbor or Scatterer::type::p2p.

inline void scatter_fwd_end(std::span<MPI_Request> requests) const

Complete a non-blocking send from the local owner to process ranks that have the index as a ghost.

This function completes the communication started by Scatterer::scatter_fwd_begin.

Parameters:

requests – [in] The MPI request handle for tracking the status of the send

template<typename T, typename F>
inline void scatter_fwd_begin(std::span<const T> local_data, std::span<T> local_buffer, std::span<T> remote_buffer, F pack_fn, std::span<MPI_Request> requests, Scatterer::type type = type::neighbor) const

Scatter data associated with owned indices to ghosting ranks.

Note

This function is intended for advanced usage, and in particular when using CUDA/device-aware MPI.

Parameters:

  • local_data – [in] All data associated with owned indices. Size is size_local() from the IndexMap used to create the scatterer, multiplied by the block size. The data for each index is blocked.

  • local_buffer – [in] Working buffer. The required size is given by Scatterer::local_buffer_size.

  • remote_buffer – [out] Working buffer. The required size is given by Scatterer::remote_buffer_size.

  • pack_fn – [in] Function to pack data from local_data into the send buffer. It is passed as an argument to support CUDA/device-aware MPI.

  • requests – [in] The MPI request handle for tracking the status of the send

  • type – [in] The type of MPI communication pattern used by the Scatterer, either Scatterer::type::neighbor or Scatterer::type::p2p.

template<typename T, typename F>
inline void scatter_fwd_end(std::span<const T> remote_buffer, std::span<T> remote_data, F unpack_fn, std::span<MPI_Request> requests) const

Complete a non-blocking send from the local owner to process ranks that have the index as a ghost, and unpack received buffer into remote data.

This function completes the communication started by Scatterer::scatter_fwd_begin.

Parameters:

  • remote_buffer – [in] Working buffer, same used in Scatterer::scatter_fwd_begin.

  • remote_data – [out] Received data associated with the ghost indices. The order follows the order of the ghost indices in the IndexMap used to create the scatterer. The size equal to the number of ghosts in the index map multiplied by the block size. The data for each index is blocked.

  • unpack_fn – [in] Function to unpack the received buffer into remote_data. It is passed as an argument to support CUDA/device-aware MPI.

  • requests – [in] The MPI request handle for tracking the status of the send

template<typename T>
inline void scatter_fwd(std::span<const T> local_data, std::span<T> remote_data) const

Scatter data associated with owned indices to ghosting ranks.

Parameters:

  • local_data – [in] All data associated with owned indices. Size is size_local() from the IndexMap used to create the scatterer, multiplied by the block size. The data for each index is blocked

  • remote_data – [out] Received data associated with the ghost indices. The order follows the order of the ghost indices in the IndexMap used to create the scatterer. The size equal to the number of ghosts in the index map multiplied by the block size. The data for each index is blocked.

template<typename T>
inline void scatter_rev_begin(std::span<const T> send_buffer, std::span<T> recv_buffer, std::span<MPI_Request> requests, Scatterer::type type = type::neighbor) const

Start a non-blocking send of ghost data to ranks that own the data.

The communication is completed by calling Scatterer::scatter_rev_end. The send and receive buffers should not be changed until after Scatterer::scatter_rev_end has been called.

The buffer must not be accessed or changed until after a call to Scatterer::scatter_fwd_end.

Parameters:

  • send_buffer – [in] Data associated with each ghost index. This data is sent to process that owns the index. It must not be changed until after a call to Scatterer::scatter_ref_end.

  • recv_buffer – Buffer used for the received data. The position of owned indices in the buffer is given by Scatterer::local_indices. Scatterer::local_displacements()[i] is the location of the first entry in recv_buffer received from neighbourhood rank i. The number of entries received from neighbourhood rank i is Scatterer::local_displacements()[i + 1] - Scatterer::local_displacements()[i]. recv_buffer[j] is the data associated with the index Scatterer::local_indices()[j] in the index map.

  • requests – The MPI request handle for tracking the status of the non-blocking communication

  • type – [in] The type of MPI communication pattern used by the Scatterer, either Scatterer::type::neighbor or Scatterer::type::p2p.

inline void scatter_rev_end(std::span<MPI_Request> request) const

End the reverse scatter communication.

This function must be called after Scatterer::scatter_rev_begin. The buffers passed to Scatterer::scatter_rev_begin must not be modified until after the function has been called.

Parameters:

request – [in] The handle used when calling Scatterer::scatter_rev_begin

template<typename T, typename F>
inline void scatter_rev_begin(std::span<const T> remote_data, std::span<T> remote_buffer, std::span<T> local_buffer, F pack_fn, std::span<MPI_Request> request, Scatterer::type type = type::neighbor) const

Scatter data associated with ghost indices to owning ranks.

Note

This function is intended for advanced usage, and in particular when using CUDA/device-aware MPI.

Template Parameters:

  • T – Data type to send.

  • F – Pack function.

Parameters:

  • remote_data – [in] Received data associated with the ghost indices. The order follows the order of the ghost indices in the IndexMap used to create the scatterer. The size equal to the number of ghosts in the index map multiplied by the block size. The data for each index is blocked.

  • remote_buffer – [out] Working buffer. The requires size is given by Scatterer::remote_buffer_size.

  • local_buffer – [out] Working buffer. The requires size is given by Scatterer::local_buffer_size.

  • pack_fn – [in] Function to pack data from local_data into the send buffer. It is passed as an argument to support CUDA/device-aware MPI.

  • request – MPI request handles for tracking the status of the non-blocking communication.

  • type – [in] Type of MPI communication pattern used by the Scatterer, either Scatterer::type::neighbor or Scatterer::type::p2p.

template<typename T, typename F, typename BinaryOp>
inline void scatter_rev_end(std::span<const T> local_buffer, std::span<T> local_data, F unpack_fn, BinaryOp op, std::span<MPI_Request> request)

End the reverse scatter communication, and unpack the received local buffer into local data.

This function must be called after Scatterer::scatter_rev_begin. The buffers passed to Scatterer::scatter_rev_begin must not be modified until after the function has been called.

Parameters:

  • local_buffer – [in] Working buffer. Same buffer should be used in Scatterer::scatter_rev_begin.

  • local_data – [out] All data associated with owned indices. Size is size_local() from the IndexMap used to create the scatterer, multiplied by the block size. The data for each index is blocked.

  • unpack_fn – [in] Function to unpack the receive buffer into local_data. It is passed as an argument to support CUDA/device-aware MPI.

  • op – [in] The reduction operation when accumulating received values. To add the received values use std::plus<T>().

  • request – [in] The handle used when calling Scatterer::scatter_rev_begin

template<typename T, typename BinaryOp>
inline void scatter_rev(std::span<T> local_data, std::span<const T> remote_data, BinaryOp op)

Scatter data associated with ghost indices to ranks that own the indices.

inline std::int32_t local_buffer_size() const noexcept

Size of buffer for local data (owned and shared) used in forward and reverse communication.

Returns:

The required buffer size

inline std::int32_t remote_buffer_size() const noexcept

Buffer size for remote data (ghosts) used in forward and reverse communication.

Returns:

The required buffer size

inline const std::vector<std::int32_t> &local_indices() const noexcept

Return a vector of local indices (owned) used to pack/unpack local data. These indices are grouped by neighbor process (process for which an index is a ghost).

inline const std::vector<std::int32_t> &remote_indices() const noexcept

Return a vector of remote indices (ghosts) used to pack/unpack ghost data. These indices are grouped by neighbor process (ghost owners).

inline int bs() const noexcept

The number values (block size) to send per index in the common::IndexMap use to create the scatterer.

Returns:

The block size

inline std::vector<MPI_Request> create_request_vector(Scatterer::type type = type::neighbor)

Create a vector of MPI_Requests for a given Scatterer::type.

Returns:

A vector of MPI requests

class TimeLogger
#include <TimeLogger.h>

Time logger maintaining data collected by Timer, if registered.

Note

This is a monotstate, i.e. the data members are static and thus timings are aggregated into a single map.

Public Functions

void register_timing(std::string task, std::chrono::duration<double, std::ratio<1>> wall)

Register timing (for later summary)

Table timing_table() const

Return a summary of timings and tasks in a Table.

void list_timings(MPI_Comm comm, Table::Reduction reduction) const

List a summary of timings and tasks. Reduction type is printed.

Parameters:

  • comm – MPI Communicator

  • reduction – Reduction type (min, max or average)

std::pair<int, std::chrono::duration<double, std::ratio<1>>> timing(std::string task) const

Return timing.

Parameters:

task – [in] The task name to retrieve the timing for

Returns:

Values (count, total wall time) for given task.

std::map<std::string, std::pair<int, std::chrono::duration<double, std::ratio<1>>>> timings() const

Logged elapsed times.

Returns:

Elapsed [task id: (count, total wall time)].

Public Static Functions

static TimeLogger &instance()

Singleton access.

Returns:

Unique time logger object.

template<typename T = std::chrono::high_resolution_clock>
class Timer
#include <Timer.h>

Timer for measuring and logging elapsed time durations.

The basic usage is 

Timer timer("Assembling over cells");
The timer is started at construction and timing ends when the timer is destroyed (goes out-of-scope). The timer can be started (reset) and stopped explicitly by 
timer.start();
 /* .... */
timer.stop();
A summary of registered elapsed times can be printed by calling: 
list_timings();
Registered elapsed times are logged when (1) the timer goes out-of-scope or (2) Timer::flush() is called.

Public Functions

inline Timer(std::optional<std::string> task = std::nullopt)

Create and start timer.

Elapsed time is optionally registered in the logger when the Timer destructor is called.

Parameters:

task – [in] Name used to registered the elapsed time in the logger. If no name is set, the elapsed time is not registered in the logger.

inline ~Timer()

If timer is still running, it is stopped. Elapsed time is registered in the logger.

inline void start()

Reset elapsed time and (re-)start timer.

template<typename Period = std::ratio<1>>
inline std::chrono::duration<double, Period> elapsed() const

Elapsed time since time has been started.

Default duration unit is seconds.

Returns:

Elapsed time duration.

template<typename Period = std::ratio<1>>
inline std::chrono::duration<double, Period> stop()

Stop timer and return elapsed time.

Default duration unit is seconds.

Returns:

Elapsed time duration.

inline void resume()

Resume a stopped timer.

Does nothing if timer has not been stopped.

inline void flush()

Flush timer duration to the logger.

An instance of a timer can be flushed to the logger only once. Subsequent calls will have no effect and will not trigger any logging.

Pre:

Timer must have been stopped before flushing.