# Efficient and Sound Differentiable Programming in a Functional Array-Processing Language

arXiv: https://arxiv.org/abs/2212.10307 · HTML: https://mlsanity.com/paper/2212.10307/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 14 of its 87 references that are in the map.
- First posted on arXiv 20 Dec 2022.

## Builds on (most cited first, 14 of 14)

- [Automatic differentiation in machine learning: a survey](https://mlsanity.com/paper/1502.05767/) (2015)
- [Glow: Graph Lowering Compiler Techniques for Neural Networks](https://mlsanity.com/paper/1805.00907/) (2018)
- [The simple essence of automatic differentiation](https://mlsanity.com/paper/1804.00746/) (2018)
- [Demystifying Differentiable Programming: Shift/Reset the Penultimate Backpropagator](https://mlsanity.com/paper/1803.10228/) (2018)
- [Correctness of Automatic Differentiation via Diffeologies and Categorical Gluing](https://mlsanity.com/paper/2001.02209/) (2020)
- [A Simple Differentiable Programming Language](https://mlsanity.com/paper/1911.04523/) (2019)
- [Optimizing Tensor Programs on Flexible Storage](https://mlsanity.com/paper/2210.06267/) (2022)
- [Efficient Differentiable Programming in a Functional Array-Processing Language](https://mlsanity.com/paper/1806.02136/) (2018)
- [Functional Collection Programming with Semi-Ring Dictionaries](https://mlsanity.com/paper/2103.06376/) (2021)
- [On the Versatility of Open Logical Relations: Continuity, Automatic Differentiation, and a Containment Theorem](https://mlsanity.com/paper/2002.08489/) (2020)
- [Backpropagation in the Simply Typed Lambda-calculus with Linear Negation](https://mlsanity.com/paper/1909.13768/) (2019)
- [Multi-layer Optimizations for End-to-End Data Analytics](https://mlsanity.com/paper/2001.03541/) (2020)
- [Stream Fusion, to Completeness](https://mlsanity.com/paper/1612.06668/) (2016)
- [A Differential-form Pullback Programming Language for Higher-order Reverse-mode Automatic Differentiation](https://mlsanity.com/paper/2002.08241/) (2020)

## Abstract (arXiv)

Automatic differentiation (AD) is a technique for computing the derivative of a function represented by a program. This technique is considered as the de-facto standard for computing the differentiation in many machine learning and optimisation software tools. Despite the practicality of this technique, the performance of the differentiated programs, especially for functional languages and in the presence of vectors, is suboptimal. We present an AD system for a higher-order functional array-processing language. The core functional language underlying this system simultaneously supports both source-to-source forward-mode AD and global optimisations such as loop transformations. In combination, gradient computation with forward-mode AD can be as efficient as reverse mode, and the Jacobian matrices required for numerical algorithms such as Gauss-Newton and Levenberg-Marquardt can be efficiently computed.