# Understanding Automatic Differentiation Pitfalls

arXiv: https://arxiv.org/abs/2305.07546 · HTML: https://mlsanity.com/paper/2305.07546/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 59 references that are in the map.
- First posted on arXiv 12 May 2023.

## Builds on (most cited first, 9 of 9)

- [Automatic differentiation in machine learning: a survey](https://mlsanity.com/paper/1502.05767/) (2015)
- [A mathematical model for automatic differentiation in machine learning](https://mlsanity.com/paper/2006.02080/) (2020)
- [Instead of Rewriting Foreign Code for Machine Learning, Automatically Synthesize Fast Gradients](https://mlsanity.com/paper/2010.01709/) (2020)
- [On Correctness of Automatic Differentiation for Non-Differentiable Functions](https://mlsanity.com/paper/2006.06903/) (2020)
- [Demystifying Differentiable Programming: Shift/Reset the Penultimate Backpropagator](https://mlsanity.com/paper/1803.10228/) (2018)
- [Correctness of Automatic Differentiation via Diffeologies and Categorical Gluing](https://mlsanity.com/paper/2001.02209/) (2020)
- [ADEV: Sound Automatic Differentiation of Expected Values of Probabilistic Programs](https://mlsanity.com/paper/2212.06386/) (2022)
- [You Only Linearize Once: Tangents Transpose to Gradients](https://mlsanity.com/paper/2204.10923/) (2022)
- [On the Correctness of Automatic Differentiation for Neural Networks with Machine-Representable Parameters](https://mlsanity.com/paper/2301.13370/) (2023)

## Built on by (most cited first, 1 of 1)

- [Correcting Auto-Differentiation in Neural-ODE Training](https://mlsanity.com/paper/2306.02192/) (2023)

## Abstract (arXiv)

Automatic differentiation, also known as backpropagation, AD, autodiff, or algorithmic differentiation, is a popular technique for computing derivatives of computer programs accurately and efficiently. Sometimes, however, the derivatives computed by AD could be interpreted as incorrect. These pitfalls occur systematically across tools and approaches. In this paper we broadly categorize problematic usages of AD and illustrate each category with examples such as chaos, time-averaged oscillations, discretizations, fixed-point loops, lookup tables, and linear solvers. We also review debugging techniques and their effectiveness in these situations. With this article we hope to help readers avoid unexpected behavior, detect problems more easily when they occur, and have more realistic expectations from AD tools.