Summary of Features

- Introduction
- 1 First-order Tangent Mode
- 1.1 Generic first-order scalar tangents
- 1.2 Generic first-order vector tangents
- 1.3 Preaccumulation through use of expression templates
- 2 First-order Adjoint Mode
- 2.1 Generic first-order scalar adjoints
- 2.2 Generic first-order vector adjoints
- 2.3 Global “blob” tape
- 2.4 Global “chunk” tape
- 2.5 Global “file” tape (v3.2)
- 2.6 Distinct data types for values, partial derivatives and adjoints with all kinds of tapes (v3.2)
- 2.7 (Thread-)local tapes
- 2.8 Preaccumulation through use of expression templates
- 2.9 Repeated evaluation of tape
- 2.10 Adjoint callback interface
- 2.11 User-defined local Jacobians interface
- 2.12 User-driven preaccumulation of local Jacobians (v3.2)
- 2.13 Multiple adjoint vectors for a single tape (v3.2)
- 2.14 Modulo Adjoint Propagation (v3.3)
- 2.15 Sparse Tape Interpretation (v3.3)
- 3 Second- and Higher-order Tangent Mode
- 3.1 Generic second-order scalar tangents
- 3.2 Generic second-order vector tangents
- 3.3 Other generic second-order tangents
- 3.4 Generic third- and higher-order tangents
- 4 Second- and Higher-order Adjoint Mode

- checkpointing
- symbolic adjoints of numerical methods
- combinations of tangent and adjoint modes
- preaccumulation of local derivative tensors
- creation of collections of domain-specific higher-level elemental functions

Instantiation of the generic first-order vector tangent data type with the generic first-order scalar tangent data type over a non-derivative base data type yields a second-order tangent data type. Similarly, instantiation of the generic first-order scalar tangent data type with the generic first-order vector tangent data type over a non-derivative base data type yields a second-order tangent data type.

Instantiation of the generic first-order scalar adjoint data type with the generic first-order scalar tangent data type over a non-derivative base data type yields a second-order scalar adjoint data type. It enables the computation of scalar projections of the Hessian $\nabla^2 f \in D^{m \times n \times n}$ in its image and one of its domain dimensions as $$ \mathbf{x}_{(1)}^{(2)}=\langle \mathbf{y}_{(1)},\nabla^2 f,\mathbf{x}^{(2)}\rangle \equiv \left(\mathbf{y}_{(1)}^T \cdot \left [\nabla^2 f \right ]_*^{j,*} \cdot \mathbf{x}^{(2)} \right)_{j=0,\ldots,n-1} \in D^n $$ for $\mathbf{y}_{(1)} \in D^m$ and $\mathbf{x}^{(2)} \in D^n$ over arbitrary base data types $D$. The computational cost of accumulating the whole Hessian over $D$ becomes $O(m\cdot n) \cdot \text{Cost}(f)$ with $\mathbf{y}_{(1)}$ and $\mathbf{x}^{(2)}$ ranging over the Cartesian basis vectors in $D^m$ and $D^n$, respectively.

Instantiation of the generic first-order vector adjoint data type with the generic first-order vector tangent data type over a non-derivative base data type yields a second-order vector adjoint data type. It enables the computation of vector projections of the Hessian $\nabla^2 f \in D^{m \times n \times n}$ in its image and one of its domain dimensions as $$ X_{(1)}^{(2)}=\langle Y_{(1)},\nabla^2 f,X^{(2)}\rangle \equiv \left(Y_{(1)}^T \cdot \left [\nabla^2 f \right ]_*^{j,*} \cdot X^{(2)} \right)_{j=0,\ldots,n-1} \in D^{l_1 \times n \times l_2} $$ for $Y_{(1)} \in D^{m \times l_1}$ and $X^{(2)} \in D^{n \times l_2}$ over arbitrary base data types $D$. The computational cost of accumulating the whole Hessian over $D$ remains equal to $O(m\cdot n) \cdot \text{Cost}(f)$ with $Y_{(1)}$ and $X^{(2)}$ set equal to the identities in $D^m$ and $D^n$, respectively.

Instantiation of the generic first-order vector adjoint data type with the generic first-order scalar tangent data type over a non-derivative base data type yields a second-order adjoint data type. Similarly, instantiation of the generic first-order scalar adjoint data type with the generic first-order vector tangent data type over a non-derivative base data type yields a second-order adjoint data type.

Symmetry of the Hessian in its two domain dimensions yields the following additional second-order adjoint data types: instantiation of a generic first-order tangent data type with a generic first-order adjoint data type over a non-derivative base data type yields a second-order adjoint data type for computing $$ \langle \mathbf{y}_{(2)}^{(1)},\nabla^2 f,\mathbf{x}^{(1)}\rangle \equiv \left(\mathbf{y}_{(2)}^{(1)^T} \cdot \left [\nabla^2 f \right ]_*^{j,*} \cdot \mathbf{x}^{(1)} \right)_{j=0,\ldots,n-1} \in D^n. $$ Similarly, instantiation of a generic first-order adjoint data type with a generic first-order adjoint data type over a non-derivative base data type yields a second-order adjoint data type for computing $$ \langle \mathbf{x}_{(1,2)},\mathbf{y}_{(1)},\nabla^2 f\rangle \equiv \left(\mathbf{y}_{(1)}^T \cdot \left [\nabla^2 f \right ]_*^{j,*} \cdot \mathbf{x}_{(1,2)} \right)_{j=0,\ldots,n-1} \in D^n. $$