# Deep Learning as the Disciplined Construction of Tame Objects

arXiv: https://arxiv.org/abs/2509.18025 · HTML: https://mlsanity.com/paper/2509.18025/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 117 references that are in the map.
- First posted on arXiv 22 Sep 2025.

## Builds on (most cited first, 21 of 21)

- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [Optimization Methods for Large-Scale Machine Learning](https://mlsanity.com/paper/1606.04838/) (2016)
- [Activation Functions in Deep Learning: A Comprehensive Survey and Benchmark](https://mlsanity.com/paper/2109.14545/) (2021)
- [Activation Functions: Comparison of trends in Practice and Research for Deep Learning](https://mlsanity.com/paper/1811.03378/) (2018)
- [Stochastic subgradient method converges on tame functions](https://mlsanity.com/paper/1804.07795/) (2018)
- [Conservative set valued fields, automatic differentiation, stochastic gradient method and deep learning](https://mlsanity.com/paper/1909.10300/) (2019)
- [Stochastic Methods for Composite and Weakly Convex Optimization Problems](https://mlsanity.com/paper/1703.08570/) (2017)
- [A mathematical model for automatic differentiation in machine learning](https://mlsanity.com/paper/2006.02080/) (2020)
- [Convergence of constant step stochastic gradient descent for non-smooth non-convex functions](https://mlsanity.com/paper/2005.08513/) (2020)
- [Three Decades of Activations: A Comprehensive Survey of 400 Activation Functions for Neural Networks](https://mlsanity.com/paper/2402.09092/) (2024)
- [On Correctness of Automatic Differentiation for Non-Differentiable Functions](https://mlsanity.com/paper/2006.06903/) (2020)
- [Active manifolds, stratifications, and convergence to local minima in nonsmooth optimization](https://mlsanity.com/paper/2108.11832/) (2021)
- [Provably Correct Automatic Subdifferentiation for Qualified Programs](https://mlsanity.com/paper/1809.08530/) (2018)
- [Curiosities and counterexamples in smooth convex optimization](https://mlsanity.com/paper/2001.07999/) (2020)
- [Newton acceleration on manifolds identified by proximal-gradient methods](https://mlsanity.com/paper/2012.12936/) (2020)
- [SAD Neural Networks: Divergent Gradient Flows and Asymptotic Optimality via o-minimal Structures](https://mlsanity.com/paper/2505.09572/) (2025)
- [The structure of conservative gradient fields](https://mlsanity.com/paper/2101.00699/) (2021)
- [Convergence of Descent Optimization Algorithms under Polyak-L ojasiewicz-Kurdyka Conditions](https://mlsanity.com/paper/2407.00812/) (2024)
- [Measurability in the Fundamental Theorem of Statistical Learning](https://mlsanity.com/paper/2410.10243/) (2024)
- [Certifying Global Optimality of AC-OPF Solutions via sparse polynomial optimization](https://mlsanity.com/paper/2109.10005/) (2021)
- [Piecewise Polynomial Regression of Tame Functions via Integer Programming](https://mlsanity.com/paper/2311.13544/) (2023)

## Built on by (most cited first, 3 of 3)

- [Stochastic Penalty-Barrier Method for Constrained Machine Learning](https://mlsanity.com/paper/2605.18618/) (2026)
- [Fast approximation and learning of binary classification tasks in o-minimal structures using ReLU neural networks](https://mlsanity.com/paper/2607.01266/) (2026)
- [Lipschitzian SLLNs for random functions](https://mlsanity.com/paper/2607.20411/) (2026)

## Abstract (arXiv)

One can see deep-learning models as compositions of functions within the so-called tame geometry. In this expository note, we give an overview of some topics at the interface of tame geometry (also known as o-minimality), optimization theory, and deep learning theory and practice. To do so, we gradually introduce the concepts and tools used to build convergence guarantees for stochastic gradient descent in a general nonsmooth nonconvex, but tame, setting. This illustrates some ways in which tame geometry is a natural mathematical framework for the study of AI systems, especially within Deep Learning.