# Born Discrete, Made Smooth: Variational Formulation of Shallow Neural Networks

arXiv: https://arxiv.org/abs/2607.02003 · HTML: https://mlsanity.com/paper/2607.02003/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 18 of its 35 references that are in the map.
- First posted on arXiv 2 Jul 2026.

## Builds on (most cited first, 18 of 18)

- [Neural Tangent Kernel: Convergence and Generalization in Neural Networks](https://mlsanity.com/paper/1806.07572/) (2018)
- [Reconciling modern machine learning practice and the bias-variance trade-off](https://mlsanity.com/paper/1812.11118/) (2018)
- [A Convergence Theory for Deep Learning via Over-Parameterization](https://mlsanity.com/paper/1811.03962/) (2018)
- [Gradient Descent Finds Global Minima of Deep Neural Networks](https://mlsanity.com/paper/1811.03804/) (2018)
- [A Mean Field View of the Landscape of Two-Layers Neural Networks](https://mlsanity.com/paper/1804.06561/) (2018)
- [On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transport](https://mlsanity.com/paper/1805.09545/) (2018)
- [Implicit Regularization in Deep Matrix Factorization](https://mlsanity.com/paper/1905.13655/) (2019)
- [Breaking the Curse of Dimensionality with Convex Neural Networks](https://mlsanity.com/paper/1412.8690/) (2014)
- [Implicit Regularization in Deep Learning May Not Be Explainable by Norms](https://mlsanity.com/paper/2005.06398/) (2020)
- [A Function Space View of Bounded Norm Infinite Width ReLU Nets: The Multivariate Case](https://mlsanity.com/paper/1910.01635/) (2019)
- [A Mean-Field Optimal Control Formulation of Deep Learning](https://mlsanity.com/paper/1807.01083/) (2018)
- [Machine Learning from a Continuous Viewpoint](https://mlsanity.com/paper/1912.12777/) (2019)
- [Learning Multi-Index Models with Neural Networks via Mean-Field Langevin Dynamics](https://mlsanity.com/paper/2408.07254/) (2024)
- [Improved Particle Approximation Error for Mean Field Neural Networks](https://mlsanity.com/paper/2405.15767/) (2024)
- [On the existence of minimizers in shallow residual ReLU neural network optimization landscapes](https://mlsanity.com/paper/2302.14690/) (2023)
- [Generalization of Scaled Deep ResNets in the Mean-Field Regime](https://mlsanity.com/paper/2403.09889/) (2024)
- [Convergence of drift-diffusion PDEs arising as Wasserstein gradient flows of convex functions](https://mlsanity.com/paper/2507.12385/) (2025)
- [Sobolev acceleration for neural networks](https://mlsanity.com/paper/2509.19773/) (2025)

## Abstract (arXiv)

Although neural networks are remarkably effective, their underlying optimization principles remain theoretically elusive, often characterized by non-convex landscapes and stochastic heuristics. In this work, we propose a paradigm shift by replacing the discrete training problem of shallow neural networks with a well-posed continuum variational surrogate. We identify a family of λ-convex functionals over parameter densities in weighted Sobolev spaces and prove that these variational problems are globally well-posed, stable, and exhibit unexpected almost C3 regularity. Unlike existing Wasserstein-based or Mean-Field approaches, which often face limited regularity and discretization challenges, our formulation provides direct access to elliptic regularity and convex analysis. This allows us to prove that the optimal parameter density can be obtained by solving a single linear system, bypassing iterative optimization entirely. We establish explicit generalization error controls at a rate of 1/α relative to the regularization parameter, and prove that finite-width networks of size N achieve the continuum optimum at an O(1/N) rate. This perspective bridges the gap between the Neural Tangent Kernel (NTK) and feature-learning regimes, providing a principled framework for understanding over-parameterization through the lens of variational calculus.