# Decomposing reverse-mode automatic differentiation

arXiv: https://arxiv.org/abs/2105.09469 · HTML: https://mlsanity.com/paper/2105.09469/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 5 of its 12 references that are in the map.
- First posted on arXiv 20 May 2021.

## Builds on (most cited first, 5 of 5)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Automatic differentiation in machine learning: a survey](https://mlsanity.com/paper/1502.05767/) (2015)
- [The simple essence of automatic differentiation](https://mlsanity.com/paper/1804.00746/) (2018)
- [Getting to the Point. Index Sets and Parallelism-Preserving Autodiff for Pointful Array Programming](https://mlsanity.com/paper/2104.05372/) (2021)
- [DiffSharp: An AD Library for .NET Languages](https://mlsanity.com/paper/1611.03423/) (2016)

## Built on by (most cited first, 4 of 4)

- [Efficient and Modular Implicit Differentiation](https://mlsanity.com/paper/2105.15183/) (2021)
- [Equinox: neural networks in JAX via callable PyTrees and filtered transformations](https://mlsanity.com/paper/2111.00254/) (2021)
- [Fast Finite Width Neural Tangent Kernel](https://mlsanity.com/paper/2206.08720/) (2022)
- [Neural Bregman Divergences for Distance Learning](https://mlsanity.com/paper/2206.04763/) (2022)

## Abstract (arXiv)

We decompose reverse-mode automatic differentiation into (forward-mode) linearization followed by transposition. Doing so isolates the essential difference between forward- and reverse-mode AD, and simplifies their joint implementation. In particular, once forward-mode AD rules are defined for every primitive operation in a source language, only linear primitives require an additional transposition rule in order to arrive at a complete reverse-mode AD implementation. This is how reverse-mode AD is written in JAX and Dex.