mlsanityData to 7 Oct 2026

A mathematical model for automatic differentiation in machine learning

arXiv 2006.02080 ↗First posted 3 Jun 2020cs.LG math.OC stat.MLcited by 44

Not on the map. Why?

Abstract (arXiv)

Automatic differentiation, as implemented today, does not have a simple mathematical model adapted to the needs of modern machine learning. In this work we articulate the relationships between differentiation of programs as implemented in practice and differentiation of nonsmooth functions. To this end we provide a simple class of functions, a nonsmooth calculus, and show how they apply to stochastic approximation methods. We also evidence the issue of artificial critical points created by algorithmic differentiation and show how usual methods avoid these points with probability one.

Key facts. Cited by 44 papers in the map (data through 7 Oct 2026). First posted on arXiv 3 Jun 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 8; 2022: 9; 2023: 10; 2024: 7; 2025: 4; 2026: 6.

44 papers.

Sort:

  1. 2021Nonsmooth Implicit Differentiation for Machine Learning and Optimizationcited by 43
  2. 2022Training invariances and the low-rank phenomenon: beyond linear networkscited by 33
  3. 2021A General Descent Aggregation Framework for Gradient-based Bi-level Optimizationcited by 28
  4. 2022Automatic differentiation of nonsmooth iterative algorithmscited by 25
  5. 2021Implicit differentiation for fast hyperparameter selection in non-smooth convex learningcited by 22
  6. 2022Flexible Differentiable Optimization via Model Transformationscited by 16
  7. 2023Tensor-Compressed Back-Propagation-Free Training for (Physics-Informed) Neural Networkscited by 12
  8. 2023The Feature Speed Formula: a flexible approach to scale hyper-parameters of deep neural networkscited by 11
  9. 2022Lifted Bregman Training of Neural Networkscited by 10
  10. 2021Stochastic Subgradient Descent Escapes Active Strict Saddles on Weakly Convex Functionscited by 9
  11. 2025Non-convergence to the optimal risk for Adam and stochastic gradient descent optimization in the training of deep neural networkscited by 6
  12. 2021A Gradient Sampling Algorithm for Stratified Maps with Applications to Topological Data Analysiscited by 5
32 more
  1. 2022Differentiating Nonsmooth Solutions to Parametric Monotone Inclusion Problemscited by 5
  2. 2023Adam-family Methods with Decoupled Weight Decay in Deep Learningcited by 5
  3. 2024Poor Man's Training on MCUs: A Memory-Efficient Quantized Back-Propagation-Free Approachcited by 5
  4. 2021The structure of conservative gradient fieldscited by 4
  5. 2022On the complexity of nonsmooth automatic differentiationcited by 4
  6. 2022Fixed-Point Automatic Differentiation of Forward--Backward Splitting Algorithms for Partly Smooth Functionscited by 4
  7. 2021Second-order step-size tuning of SGD for non-convex optimizationcited by 3
  8. 2023On the Correctness of Automatic Differentiation for Neural Networks with Machine-Representable Parameterscited by 3
  9. 2023Generalizing Adam to Manifolds for Efficiently Training Transformerscited by 3
  10. 2023Stochastic Subgradient Methods with Guaranteed Global Stability in Nonsmooth Nonconvex Optimizationcited by 3
  11. 2024Nonsmooth Implicit Differentiation: Deterministic and Stochastic Convergence Ratescited by 2
  12. 2024Developing Lagrangian-based Methods for Nonsmooth Nonconvex Optimizationcited by 2
  13. 2025Understanding Nonlinear Implicit Bias via Region Counts in Input Spacecited by 2
  14. 2025Deep Learning as the Disciplined Construction of Tame Objectscited by 2
  15. 2022Path differentiability of ODE flowscited by 1
  16. 2022Perturbed Iterate SGD for Lipschitz Continuous Loss Functions with Numerical Error and Adaptive Step Sizescited by 1
  17. 2023ωPAP Spaces: Reasoning Denotationally About Higher-Order, Recursive Probabilistic and Differentiable Programscited by 1
  18. 2023Understanding Automatic Differentiation Pitfallscited by 1
  19. 2023Auto-Differentiation of Relational Computations for Very Large Scale Machine Learningcited by 1
  20. 2023Piecewise Polynomial Regression of Tame Functions via Integer Programmingcited by 1
  21. 2024Convergence of Decentralized Stochastic Subgradient-based Methods for Nonsmooth Nonconvex functionscited by 1
  22. 2024Automatic Differentiation of Optimization Algorithms with Time-Varying Updatescited by 1
  23. 2024Hamiltonian Monte Carlo on ReLU Neural Networks is Inefficientcited by 1
  24. 2025Stochastic Sample Approximations of (Local) Moduli of Continuitycited by 1
  25. 2026The Normalized Maximum Likelihood for Regular Non-Smooth Models: Measure-Theoretic Foundations and Geometric Samplingcited by 1
  26. 2026A Blueprint for Equilibrium-Based Differentiable Continuous-Variable Thermodynamic Computingcited by 1
  27. 2021Numerical influence of ReLU'(0) on backpropagationcited by 0
  28. 2024On the numerical reliability of nonsmooth autodiff: a MaxPool case studycited by 0
  29. 2026partialCBDs: Differentiable Causal Block Diagramscited by 0
  30. 2026Dual Certified White-Box Inference for Input Convex Neural Networkscited by 0
  31. 2026Decentralized Stochastic Subgradient-type Methods with Communication Compression for Nonsmooth Nonconvex Optimizationcited by 0
  32. 2026Same Loss, Different Gradientscited by 0