# The simple essence of automatic differentiation

arXiv: https://arxiv.org/abs/1804.00746 · HTML: https://mlsanity.com/paper/1804.00746/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Apr 2018.

## Built on by (most cited first, 23 of 23)

- [Categorical Foundations of Gradient-Based Learning](https://mlsanity.com/paper/2103.01931/) (2021)
- [Category Theory in Machine Learning](https://mlsanity.com/paper/2106.07032/) (2021)
- [Position: Categorical Deep Learning is an Algebraic Theory of All Architectures](https://mlsanity.com/paper/2402.15332/) (2024)
- [On Meta-Prompting](https://mlsanity.com/paper/2312.06562/) (2023)
- [Fast Jacobian-Vector Product for Deep Networks](https://mlsanity.com/paper/2104.00219/) (2021)
- [Decomposing reverse-mode automatic differentiation](https://mlsanity.com/paper/2105.09469/) (2021)
- [On the Correctness of Automatic Differentiation for Neural Networks with Machine-Representable Parameters](https://mlsanity.com/paper/2301.13370/) (2023)
- [Deep Learning with Parametric Lenses](https://mlsanity.com/paper/2404.00408/) (2024)
- [Functorial String Diagrams for Reverse-Mode Automatic Differentiation](https://mlsanity.com/paper/2107.13433/) (2021)
- [Space-time tradeoffs of lenses and optics via higher category theory](https://mlsanity.com/paper/2209.09351/) (2022)
- [Implicit Bilevel Optimization: Differentiating through Bilevel Optimization Programming](https://mlsanity.com/paper/2302.14473/) (2023)
- [Automatic Functional Differentiation in JAX](https://mlsanity.com/paper/2311.18727/) (2023)
- [Generalized Optimization: A First Step Towards Category Theoretic Learning Theory](https://mlsanity.com/paper/2109.10262/) (2021)
- [Folding over Neural Networks](https://mlsanity.com/paper/2207.01090/) (2022)
- [Jet Functors and Weil Algebras in Automatic Differentiation: A Geometric Analysis](https://mlsanity.com/paper/2510.14342/) (2025)
- [Combinatory Adjoints and Differentiation](https://mlsanity.com/paper/2207.00847/) (2022)
- [Efficient and Sound Differentiable Programming in a Functional Array-Processing Language](https://mlsanity.com/paper/2212.10307/) (2022)
- [nablaSD: Differentiable Programming for Sparse Tensors](https://mlsanity.com/paper/2303.07030/) (2023)
- [Monadic Deep Learning](https://mlsanity.com/paper/2307.12187/) (2023)
- [Algebraic Dynamical Systems in Machine Learning](https://mlsanity.com/paper/2311.03118/) (2023)
- [Compiling to recurrent neurons](https://mlsanity.com/paper/2511.14953/) (2025)
- [Differentiate the Evaluator, Not the Program: An Efficient Runtime Representation for Neuro-Symbolic Learning](https://mlsanity.com/paper/2607.03574/) (2026)
- [Coordinate-Free Automatic Differentiation](https://mlsanity.com/paper/2610.12776/) (2026)

## Abstract (arXiv)

Automatic differentiation (AD) in reverse mode (RAD) is a central component of deep learning and other uses of large-scale optimization. Commonly used RAD algorithms such as backpropagation, however, are complex and stateful, hindering deep understanding, improvement, and parallel execution. This paper develops a simple, generalized AD algorithm calculated from a simple, natural specification. The general algorithm is then specialized by varying the representation of derivatives. In particular, applying well-known constructions to a naive representation yields two RAD algorithms that are far simpler than previously known. In contrast to commonly used RAD implementations, the algorithms defined here involve no graphs, tapes, variables, partial derivatives, or mutation. They are inherently parallel-friendly, correct by construction, and usable directly from an existing programming language with no need for new data types or programming style, thanks to use of an AD-agnostic compiler plugin.