# On the Correctness of Automatic Differentiation for Neural Networks with Machine-Representable Parameters

arXiv: https://arxiv.org/abs/2301.13370 · HTML: https://mlsanity.com/paper/2301.13370/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 61 references that are in the map.
- First posted on arXiv 31 Jan 2023.

## Builds on (most cited first, 27 of 27)

- [Neural Tangent Kernel: Convergence and Generalization in Neural Networks](https://mlsanity.com/paper/1806.07572/) (2018)
- [Automatic differentiation in machine learning: a survey](https://mlsanity.com/paper/1502.05767/) (2015)
- [Deep Learning in Neural Networks: An Overview](https://mlsanity.com/paper/1404.7828/) (2014)
- [The Expressive Power of Neural Networks: A View from the Width](https://mlsanity.com/paper/1709.02540/) (2017)
- [Caffe: Convolutional Architecture for Fast Feature Embedding](https://mlsanity.com/paper/1408.5093/) (2014)
- [Universal Approximation with Deep Narrow Networks](https://mlsanity.com/paper/1905.08539/) (2019)
- [Stochastic subgradient method converges on tame functions](https://mlsanity.com/paper/1804.07795/) (2018)
- [Minimum Width for Universal Approximation](https://mlsanity.com/paper/2006.08859/) (2020)
- [Conservative set valued fields, automatic differentiation, stochastic gradient method and deep learning](https://mlsanity.com/paper/1909.10300/) (2019)
- [A mathematical model for automatic differentiation in machine learning](https://mlsanity.com/paper/2006.02080/) (2020)
- [The simple essence of automatic differentiation](https://mlsanity.com/paper/1804.00746/) (2018)
- [Chainer: A Deep Learning Framework for Accelerating the Research Cycle](https://mlsanity.com/paper/1908.00213/) (2019)
- [On Correctness of Automatic Differentiation for Non-Differentiable Functions](https://mlsanity.com/paper/2006.06903/) (2020)
- [Correctness of Automatic Differentiation via Diffeologies and Categorical Gluing](https://mlsanity.com/paper/2001.02209/) (2020)
- [A Simple Differentiable Programming Language](https://mlsanity.com/paper/1911.04523/) (2019)
- [You Only Linearize Once: Tangents Transpose to Gradients](https://mlsanity.com/paper/2204.10923/) (2022)
- [The Multilinear Structure of ReLU Networks](https://mlsanity.com/paper/1712.10132/) (2017)
- [Provably Correct Automatic Subdifferentiation for Qualified Programs](https://mlsanity.com/paper/1809.08530/) (2018)
- [On the Versatility of Open Logical Relations: Continuity, Automatic Differentiation, and a Containment Theorem](https://mlsanity.com/paper/2002.08489/) (2020)
- [Backpropagation in the Simply Typed Lambda-calculus with Linear Negation](https://mlsanity.com/paper/1909.13768/) (2019)
- [Automatic Differentiation in PCF](https://mlsanity.com/paper/2011.03335/) (2020)
- [On the complexity of nonsmooth automatic differentiation](https://mlsanity.com/paper/2206.01730/) (2022)
- [DiffSharp: An AD Library for .NET Languages](https://mlsanity.com/paper/1611.03423/) (2016)
- [Reverse AD at Higher Types: Pure, Principled and Denotationally Correct](https://mlsanity.com/paper/2007.05283/) (2020)
- [Tangent: Automatic differentiation using source-code transformation for dynamically typed array programming](https://mlsanity.com/paper/1809.09569/) (2018)
- [Parallel Dual-Numbers Reverse AD](https://mlsanity.com/paper/2207.03418/) (2022)
- [Towards Denotational Semantics of AD for Higher-Order, Recursive, Probabilistic Languages](https://mlsanity.com/paper/2111.15456/) (2021)

## Built on by (most cited first, 3 of 3)

- [Understanding Automatic Differentiation Pitfalls](https://mlsanity.com/paper/2305.07546/) (2023)
- [On the numerical reliability of nonsmooth autodiff: a MaxPool case study](https://mlsanity.com/paper/2401.02736/) (2024)
- [Dual Certified White-Box Inference for Input Convex Neural Networks](https://mlsanity.com/paper/2605.04722/) (2026)

## Abstract (arXiv)

Recent work has shown that forward- and reverse- mode automatic differentiation (AD) over the reals is almost always correct in a mathematically precise sense. However, actual programs work with machine-representable numbers (e.g., floating-point numbers), not reals. In this paper, we study the correctness of AD when the parameter space of a neural network consists solely of machine-representable numbers. In particular, we analyze two sets of parameters on which AD can be incorrect: the incorrect set on which the network is differentiable but AD does not compute its derivative, and the non-differentiable set on which the network is non-differentiable. For a neural network with bias parameters, we first prove that the incorrect set is always empty. We then prove a tight bound on the size of the non-differentiable set, which is linear in the number of non-differentiabilities in activation functions, and give a simple necessary and sufficient condition for a parameter to be in this set. We further prove that AD always computes a Clarke subderivative even on the non-differentiable set. We also extend these results to neural networks possibly without bias parameters.