mlsanityData to 7 Oct 2026

Automatic differentiation in machine learning: a survey

arXiv 1502.05767 ↗First posted 20 Feb 2015cs.SC cs.LG stat.MLcited by 846

Not on the map. Why?

Abstract (arXiv)

Derivatives, mostly in the form of gradients and Hessians, are ubiquitous in machine learning. Automatic differentiation (AD), also called algorithmic differentiation or simply "autodiff", is a family of techniques similar to but more general than backpropagation for efficiently and accurately evaluating derivatives of numeric functions expressed as computer programs. AD is a small but established field with applications in areas including computational fluid dynamics, atmospheric sciences, and engineering design optimization. Until very recently, the fields of machine learning and AD have largely been unaware of each other and, in some cases, have independently discovered each other's results. Despite its relevance, general-purpose AD has been missing from the machine learning toolbox, a situation slowly changing with its ongoing adoption under the names "dynamic computational graphs" and "differentiable programming". We survey the intersection of AD and machine learning, cover applications where AD has direct relevance, and address the main implementation techniques. By precisely defining the main differentiation techniques and their interrelationships, we aim to bring clarity to the usage of the terms "autodiff", "automatic differentiation", and "symbolic differentiation" as these are encountered more and more in machine learning settings.

Key facts. Cited by 846 papers in the map (data through 7 Oct 2026). First posted on arXiv 20 Feb 2015.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 142; 2022: 150; 2023: 158; 2024: 153; 2025: 120; 2026: 123.

846 papers.

Sort:

By area:

  1. 2021Neural Fields in Visual Computing and Beyondcited by 539
  2. 2021Learning the solution operator of parametric partial differential equations with physics-informed DeepOnetscited by 537
  3. 2021Gradient-enhanced physics-informed neural networks for forward and inverse PDE problemscited by 220
  4. 2023Task Arithmetic in the Tangent Space: Improved Editing of Pre-Trained Modelscited by 218
  5. 2025Real-Time Execution of Action Chunking Flow Policiescited by 199
  6. 2021Adversarial Unlearning of Backdoors via Implicit Hypergradientcited by 160
  7. 2021Investigating Bi-Level Optimization for Learning and Vision from a Unified Perspective: A Survey and Beyondcited by 151
  8. 2021Hyperparameter Optimization: Foundations, Algorithms, Best Practices and Open Challengescited by 143
  9. 2021Scaling Up Influence Functionscited by 130
  10. 2023Decision-Focused Learning: Foundations, State of the Art, Benchmark and Future Opportunitiescited by 129
  11. 2021Efficient training of physics-informed neural networks via importance samplingcited by 109
  12. 2021Deep Adaptive Design: Amortizing Sequential Bayesian Experimental Designcited by 103
38 more
  1. 2022A framework for bilevel optimization that enables stochastic and global variance reduction algorithmscited by 99
  2. 2021CAN-PINN: A Fast Physics-Informed Neural Network Based on Coupled-Automatic-Numerical Differentiation Methodcited by 93
  3. 2021Long-time integration of parametric evolution equations with physics-informed DeepONetscited by 87
  4. 2021Improved architectures and training algorithms for deep operator networkscited by 85
  5. 2022Gradients without Backpropagationcited by 73
  6. 2024From PINNs to PIKANs: Recent Advances in Physics-Informed Machine Learningcited by 73
  7. 2022Formal Algorithms for Transformerscited by 70
  8. 2023Prompt-tuning latent diffusion models for inverse problemscited by 70
  9. 2021Physics-informed neural networks for solving Reynolds-averaged Navierunicodex2013Stokes equationscited by 68
  10. 2023Can Physics-Informed Neural Networks beat the Finite Element Method?cited by 68
  11. 2023Separable Physics-Informed Neural Networkscited by 64
  12. 2023Scallop: A Language for Neurosymbolic Programmingcited by 63
  13. 2021Amortized Implicit Differentiation for Stochastic Bilevel Optimizationcited by 62
  14. 2024Trace is the Next AutoDiff: Generative Optimization with Rich Feedback, Execution Traces, and LLMscited by 62
  15. 2021Learning in Sinusoidal Spaces with Physics-Informed Neural Networkscited by 61
  16. 2021Implicit Deep Adaptive Design: Policy-Based Experimental Design without Likelihoodscited by 61
  17. 2021Polyconvex anisotropic hyperelasticity with neural networkscited by 54
  18. 2021Robust Generalised Bayesian Inference for Intractable Likelihoodscited by 53
  19. 2021Rectangular Flows for Manifold Learningcited by 49
  20. 2022Derivative-Informed Neural Operator: An Efficient Framework for High-Dimensional Parametric Derivative Learningcited by 43
  21. 2024A Geometric View of Data Complexity: Efficient Local Intrinsic Dimension Estimation with Diffusion Modelscited by 41
  22. 2021Tangent Space Backpropagation for 3D Transformation Groupscited by 40
  23. 2024Adaptive Training of Grid-Dependent Physics-Informed Kolmogorov-Arnold Networkscited by 37
  24. 2021Momentum Residual Neural Networkscited by 36
  25. 2021Meta-Auto-Decoder for Solving Parametric Partial Differential Equationscited by 36
  26. 2022JAX-FLUIDS: A fully-differentiable high-order computational fluid dynamics solver for compressible two-phase flowscited by 36
  27. 2023Online Hyperparameter Optimization for Class-Incremental Learningcited by 36
  28. 2021A novel meta-learning initialization method for physics-informed neural networkscited by 35
  29. 2024A review on data-driven constitutive laws for solidscited by 35
  30. 2024Parameterized Physics-informed Neural Networks for Parameterized PDEscited by 35
  31. 2021A Trainable Spectral-Spatial Sparse Coding Model for Hyperspectral Image Restorationcited by 34
  32. 2022Tractable Dendritic RNNs for Reconstructing Nonlinear Dynamical Systemscited by 34
  33. 2022Deep Learning Methods for Partial Differential Equations and Related Parameter Identification Problemscited by 34
  34. 2022A mixed formulation for physics-informed neural networks as a potential solver for engineering problems in heterogeneous domains: comparison with finite element methodcited by 33
  35. 2021High-order Differentiable Autoencoder for Nonlinear Model Reductioncited by 32
  36. 2021Physics-informed neural networks for the shallow-water equations on the spherecited by 32
  37. 2023Learning to Warm-Start Fixed-Point Optimization Algorithmscited by 32
  38. 2024Position: Categorical Deep Learning is an Algebraic Theory of All Architecturescited by 32

50 most cited of 846 All 846 (JSON)