mlsanityData to 14 Oct 2026

Conservative set valued fields, automatic differentiation, stochastic gradient method and deep learning

arXiv 1909.10300 ↗First posted 23 Sep 2019math.OC cs.AI cs.LGcited by 72

Not on the map. Why?

Abstract (arXiv)

Modern problems in AI or in numerical analysis require nonsmooth approaches with a flexible calculus. We introduce generalized derivatives called conservative fields for which we develop a calculus and provide representation formulas. Functions having a conservative field are called path differentiable: convex, concave, Clarke regular and any semialgebraic Lipschitz continuous functions are path differentiable. Using Whitney stratification techniques for semialgebraic and definable sets, our model provides variational formulas for nonsmooth automatic differentiation oracles, as for instance the famous backpropagation algorithm in deep learning. Our differential model is applied to establish the convergence in values of nonsmooth stochastic gradient methods as they are implemented in practice.

Key facts. Cited by 72 papers in the map (data through 14 Oct 2026). First posted on arXiv 23 Sep 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 8; 2022: 8; 2023: 11; 2024: 13; 2025: 14; 2026: 18.

72 papers.

Sort:

By area:

  1. 2021Nonsmooth Implicit Differentiation for Machine Learning and Optimizationcited by 43
  2. 2022Gradient-Free Methods for Deterministic and Stochastic Nonsmooth Nonconvex Optimizationcited by 43
  3. 2025The Surprising Agreement Between Convex Optimization Theory and Learning-Rate Scheduling for Large Model Trainingcited by 33
  4. 2023Deterministic Nonsmooth Nonconvex Optimizationcited by 26
  5. 2022Automatic differentiation of nonsmooth iterative algorithmscited by 25
  6. 2021Implicit differentiation for fast hyperparameter selection in non-smooth convex learningcited by 22
  7. 2023Adam-family Methods for Nonsmooth Optimization with Convergence Guaranteescited by 22
  8. 2025Learning Regularization Functionals for Inverse Problems: A Comparative Studycited by 15
  9. 2023Properties of Discrete Sliced Wasserstein Lossescited by 12
  10. 2021Convergence analysis for gradient flows in the training of artificial neural networks with ReLU activationcited by 11
  11. 2021Stochastic Subgradient Descent Escapes Active Strict Saddles on Weakly Convex Functionscited by 9
  12. 2021Modeling Design and Control Problems Involving Neural Network Surrogatescited by 9
38 more
  1. 2024Early Directional Convergence in Deep Homogeneous Neural Networks for Small Initializationscited by 7
  2. 2025Differentiable Generalized Sliced Wasserstein Planscited by 7
  3. 2025Non-convergence to the optimal risk for Adam and stochastic gradient descent optimization in the training of deep neural networkscited by 6
  4. 2022An Inexact Augmented Lagrangian Algorithm for Training Leaky ReLU Neural Network with Group Sparsitycited by 5
  5. 2022Differentiating Nonsmooth Solutions to Parametric Monotone Inclusion Problemscited by 5
  6. 2023Adam-family Methods with Decoupled Weight Decay in Deep Learningcited by 5
  7. 2024Stability of Data-Dependent Ridge-Regularization for Inverse Problemscited by 5
  8. 2021The structure of conservative gradient fieldscited by 4
  9. 2022On the complexity of nonsmooth automatic differentiationcited by 4
  10. 2022Fixed-Point Automatic Differentiation of Forward--Backward Splitting Algorithms for Partly Smooth Functionscited by 4
  11. 2023Convergence of SGD for Training Neural Networks with Sliced Wasserstein Lossescited by 4
  12. 2023Taming Binarized Neural Networks and Mixed-Integer Programscited by 4
  13. 2026CATTO: Balancing Preferences and Confidence in Language Modelscited by 4
  14. 2023On the Correctness of Automatic Differentiation for Neural Networks with Machine-Representable Parameterscited by 3
  15. 2023Stochastic Subgradient Methods with Guaranteed Global Stability in Nonsmooth Nonconvex Optimizationcited by 3
  16. 2025On exploration of an interior mirror descent flow for stochastic nonconvex constrained problemcited by 3
  17. 2025Deep Learning as the Disciplined Construction of Tame Objectscited by 3
  18. 2026Stored in Optimizer State, Valued by Later Training: A Causal Account of Subliminal Trait Transfercited by 3
  19. 2022Stochastic Langevin Differential Inclusions with Applications to Machine Learningcited by 2
  20. 2024GD doesn't make the cut: Three ways that non-differentiability affects neural network trainingcited by 2
  21. 2024Nonsmooth Implicit Differentiation: Deterministic and Stochastic Convergence Ratescited by 2
  22. 2024Learning truly monotone operators with applications to nonlinear inverse problemscited by 2
  23. 2024Developing Lagrangian-based Methods for Nonsmooth Nonconvex Optimizationcited by 2
  24. 2024Memory-Efficient 4-bit Preconditioned Stochastic Optimizationcited by 2
  25. 2025Understanding Nonlinear Implicit Bias via Region Counts in Input Spacecited by 2
  26. 2021Stochastic Subgradient Descent on a Generic Definable Function Converges to a Minimizercited by 1
  27. 2022Path differentiability of ODE flowscited by 1
  28. 2023Designing Optimal Personalized Incentive for Traffic Routing using BIG Hype algorithmcited by 1
  29. 2023Piecewise Polynomial Regression of Tame Functions via Integer Programmingcited by 1
  30. 2024Inexact subgradient methods for semialgebraic functionscited by 1
  31. 2024Automatic Differentiation of Optimization Algorithms with Time-Varying Updatescited by 1
  32. 2025The late-stage training dynamics of (stochastic) subgradient descent on homogeneous neural networkscited by 1
  33. 2025Accelerating Optimization via Differentiable Stopping Timecited by 1
  34. 2025Stochastic Sample Approximations of (Local) Moduli of Continuitycited by 1
  35. 2026Exploiting Subgradient Sparsity in Max-Plus Neural Networkscited by 1
  36. 2026rSDNet: Unified Robust Neural Learning against Label Noise and Adversarial Attackscited by 1
  37. 2026The Normalized Maximum Likelihood for Regular Non-Smooth Models: Measure-Theoretic Foundations and Geometric Samplingcited by 1
  38. 2021Numerical influence of ReLU'(0) on backpropagationcited by 0

50 most cited of 72 All 72 (JSON)