# You Only Linearize Once: Tangents Transpose to Gradients

arXiv: https://arxiv.org/abs/2204.10923 · HTML: https://mlsanity.com/paper/2204.10923/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Apr 2022.

## Built on by (most cited first, 10 of 10)

- [Fast Finite Width Neural Tangent Kernel](https://mlsanity.com/paper/2206.08720/) (2022)
- [Probabilistic Programming with Programmable Variational Inference](https://mlsanity.com/paper/2406.15742/) (2024)
- [On the Correctness of Automatic Differentiation for Neural Networks with Machine-Representable Parameters](https://mlsanity.com/paper/2301.13370/) (2023)
- [ωPAP Spaces: Reasoning Denotationally About Higher-Order, Recursive Probabilistic and Differentiable Programs](https://mlsanity.com/paper/2302.10636/) (2023)
- [Automatic Functional Differentiation in JAX](https://mlsanity.com/paper/2311.18727/) (2023)
- [Understanding Automatic Differentiation Pitfalls](https://mlsanity.com/paper/2305.07546/) (2023)
- [nablaSD: Differentiable Programming for Sparse Tensors](https://mlsanity.com/paper/2303.07030/) (2023)
- [Bidirectional Reactive Programming for Machine Learning](https://mlsanity.com/paper/2311.16977/) (2023)
- [Robust Automatic Differentiation of Square-Root Kalman Filters via Gramian Differentials](https://mlsanity.com/paper/2603.13559/) (2026)
- [Archimedean Copula Inference via Taylor-Mode AD](https://mlsanity.com/paper/2605.23134/) (2026)

## Abstract (arXiv)

Automatic differentiation (AD) is conventionally understood as a family of distinct algorithms, rooted in two "modes" -- forward and reverse -- which are typically presented (and implemented) separately. Can there be only one? Following up on the AD systems developed in the JAX and Dex projects, we formalize a decomposition of reverse-mode AD into (i) forward-mode AD followed by (ii) unzipping the linear and non-linear parts and then (iii) transposition of the linear part. To that end, we define a (substructurally) linear type system that can prove a class of functions are (algebraically) linear. Our main results are that forward-mode AD produces such linear functions, and that we can unzip and transpose any such linear function, conserving cost, size, and linearity. Composing these three transformations recovers reverse-mode AD. This decomposition also sheds light on checkpointing, which emerges naturally from a free choice in unzipping `let` expressions. As a corollary, checkpointing techniques are applicable to general-purpose partial evaluation, not just AD. We hope that our formalization will lead to a deeper understanding of automatic differentiation and that it will simplify implementations, by separating the concerns of differentiation proper from the concerns of gaining efficiency (namely, separating the derivative computation from the act of running it backward).