# Conservative set valued fields, automatic differentiation, stochastic gradient method and deep learning

arXiv: https://arxiv.org/abs/1909.10300 · HTML: https://mlsanity.com/paper/1909.10300/

## Key facts

- Cited by 72 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Sep 2019.

## Built on by (most cited first, 30 of 72)

- [Nonsmooth Implicit Differentiation for Machine Learning and Optimization](https://mlsanity.com/paper/2106.04350/) (2021)
- [Gradient-Free Methods for Deterministic and Stochastic Nonsmooth Nonconvex Optimization](https://mlsanity.com/paper/2209.05045/) (2022)
- [The Surprising Agreement Between Convex Optimization Theory and Learning-Rate Scheduling for Large Model Training](https://mlsanity.com/paper/2501.18965/) (2025)
- [Deterministic Nonsmooth Nonconvex Optimization](https://mlsanity.com/paper/2302.08300/) (2023)
- [Automatic differentiation of nonsmooth iterative algorithms](https://mlsanity.com/paper/2206.00457/) (2022)
- [Implicit differentiation for fast hyperparameter selection in non-smooth convex learning](https://mlsanity.com/paper/2105.01637/) (2021)
- [Adam-family Methods for Nonsmooth Optimization with Convergence Guarantees](https://mlsanity.com/paper/2305.03938/) (2023)
- [Learning Regularization Functionals for Inverse Problems: A Comparative Study](https://mlsanity.com/paper/2510.01755/) (2025)
- [Properties of Discrete Sliced Wasserstein Losses](https://mlsanity.com/paper/2307.10352/) (2023)
- [Convergence analysis for gradient flows in the training of artificial neural networks with ReLU activation](https://mlsanity.com/paper/2107.04479/) (2021)
- [Stochastic Subgradient Descent Escapes Active Strict Saddles on Weakly Convex Functions](https://mlsanity.com/paper/2108.02072/) (2021)
- [Modeling Design and Control Problems Involving Neural Network Surrogates](https://mlsanity.com/paper/2111.10489/) (2021)
- [Early Directional Convergence in Deep Homogeneous Neural Networks for Small Initializations](https://mlsanity.com/paper/2403.08121/) (2024)
- [Differentiable Generalized Sliced Wasserstein Plans](https://mlsanity.com/paper/2505.22049/) (2025)
- [Non-convergence to the optimal risk for Adam and stochastic gradient descent optimization in the training of deep neural networks](https://mlsanity.com/paper/2503.01660/) (2025)
- [An Inexact Augmented Lagrangian Algorithm for Training Leaky ReLU Neural Network with Group Sparsity](https://mlsanity.com/paper/2205.05428/) (2022)
- [Differentiating Nonsmooth Solutions to Parametric Monotone Inclusion Problems](https://mlsanity.com/paper/2212.07844/) (2022)
- [Adam-family Methods with Decoupled Weight Decay in Deep Learning](https://mlsanity.com/paper/2310.08858/) (2023)
- [Stability of Data-Dependent Ridge-Regularization for Inverse Problems](https://mlsanity.com/paper/2406.12289/) (2024)
- [The structure of conservative gradient fields](https://mlsanity.com/paper/2101.00699/) (2021)
- [On the complexity of nonsmooth automatic differentiation](https://mlsanity.com/paper/2206.01730/) (2022)
- [Fixed-Point Automatic Differentiation of Forward--Backward Splitting Algorithms for Partly Smooth Functions](https://mlsanity.com/paper/2208.03107/) (2022)
- [Convergence of SGD for Training Neural Networks with Sliced Wasserstein Losses](https://mlsanity.com/paper/2307.11714/) (2023)
- [Taming Binarized Neural Networks and Mixed-Integer Programs](https://mlsanity.com/paper/2310.04469/) (2023)
- [CATTO: Balancing Preferences and Confidence in Language Models](https://mlsanity.com/paper/2601.23096/) (2026)
- [On the Correctness of Automatic Differentiation for Neural Networks with Machine-Representable Parameters](https://mlsanity.com/paper/2301.13370/) (2023)
- [Stochastic Subgradient Methods with Guaranteed Global Stability in Nonsmooth Nonconvex Optimization](https://mlsanity.com/paper/2307.10053/) (2023)
- [On exploration of an interior mirror descent flow for stochastic nonconvex constrained problem](https://mlsanity.com/paper/2507.15264/) (2025)
- [Deep Learning as the Disciplined Construction of Tame Objects](https://mlsanity.com/paper/2509.18025/) (2025)
- [Stored in Optimizer State, Valued by Later Training: A Causal Account of Subliminal Trait Transfer](https://mlsanity.com/paper/2608.20442/) (2026)

## Abstract (arXiv)

Modern problems in AI or in numerical analysis require nonsmooth approaches with a flexible calculus. We introduce generalized derivatives called conservative fields for which we develop a calculus and provide representation formulas. Functions having a conservative field are called path differentiable: convex, concave, Clarke regular and any semialgebraic Lipschitz continuous functions are path differentiable. Using Whitney stratification techniques for semialgebraic and definable sets, our model provides variational formulas for nonsmooth automatic differentiation oracles, as for instance the famous backpropagation algorithm in deep learning. Our differential model is applied to establish the convergence in values of nonsmooth stochastic gradient methods as they are implemented in practice.