# Machine Learning from a Continuous Viewpoint

arXiv: https://arxiv.org/abs/1912.12777 · HTML: https://mlsanity.com/paper/1912.12777/

## Key facts

- Cited by 50 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Dec 2019.

## Built on by (most cited first, 30 of 50)

- [Overview frequency principle/spectral bias in deep learning](https://mlsanity.com/paper/2201.07395/) (2022)
- [Noisy Recurrent Neural Networks](https://mlsanity.com/paper/2102.04877/) (2021)
- [SPINN: Sparse, Physics-based, and partially Interpretable Neural Networks for PDEs](https://mlsanity.com/paper/2102.13037/) (2021)
- [Overparameterization of deep ResNet: zero loss and mean-field analysis](https://mlsanity.com/paper/2105.14417/) (2021)
- [Operator Learning Using Random Features: A Tool for Scientific Computing](https://mlsanity.com/paper/2408.06526/) (2024)
- [PottsMGNet: A Mathematical Explanation of Encoder-Decoder Based Neural Networks](https://mlsanity.com/paper/2307.09039/) (2023)
- [Analysis of feature learning in weight-tied autoencoders via the mean field lens](https://mlsanity.com/paper/2102.08373/) (2021)
- [Exact Gap between Generalization Error and Uniform Convergence in Random Feature Models](https://mlsanity.com/paper/2103.04554/) (2021)
- [On Feature Learning in Neural Networks with Global Convergence Guarantees](https://mlsanity.com/paper/2204.10782/) (2022)
- [Model-free tracking control of complex dynamical trajectories with machine learning](https://mlsanity.com/paper/2309.11470/) (2023)
- [On the Global Convergence of Gradient Descent for multi-layer ResNets in the mean-field regime](https://mlsanity.com/paper/2110.02926/) (2021)
- [Subspace Decomposition based DNN algorithm for elliptic type multi-scale PDEs](https://mlsanity.com/paper/2112.06660/) (2021)
- [Defects of Convolutional Decoder Networks in Frequency Representation](https://mlsanity.com/paper/2210.09020/) (2022)
- [Provably convergent quasistatic dynamics for mean-field two-player zero-sum games](https://mlsanity.com/paper/2202.10947/) (2022)
- [Generalization Error of GAN from the Discriminator's Perspective](https://mlsanity.com/paper/2107.03633/) (2021)
- [On the symmetries in the dynamics of wide two-layer neural networks](https://mlsanity.com/paper/2211.08771/) (2022)
- [A Mathematical Framework for Learning Probability Distributions](https://mlsanity.com/paper/2212.11481/) (2022)
- [Stochastic Modified Flows, Mean-Field Limits and Dynamics of Stochastic Gradient Descent](https://mlsanity.com/paper/2302.07125/) (2023)
- [A blob method for inhomogeneous diffusion with applications to multi-agent control and sampling](https://mlsanity.com/paper/2202.12927/) (2022)
- [A convergence result of a continuous model of deep learning via a Lojasiewicz--Simon inequality](https://mlsanity.com/paper/2311.15365/) (2023)
- [Learning Domain-Independent Green's Function For Elliptic Partial Differential Equations](https://mlsanity.com/paper/2401.17172/) (2024)
- [Frequency Principle in Deep Learning Beyond Gradient-descent-based Training](https://mlsanity.com/paper/2101.00747/) (2021)
- [An Upper Limit of Decaying Rate with Respect to Frequency in Deep Neural Network](https://mlsanity.com/paper/2105.11675/) (2021)
- [Optimal bump functions for shallow ReLU networks: Weight decay, depth separation and the curse of dimensionality](https://mlsanity.com/paper/2209.01173/) (2022)
- [A Mathematical Explanation of UNet](https://mlsanity.com/paper/2410.04434/) (2024)
- [A Deterministic Sampling Method via Maximum Mean Discrepancy Flow with Adaptive Kernel](https://mlsanity.com/paper/2111.10722/) (2021)
- [Conservative SPDEs as fluctuating mean field limits of stochastic gradient descent](https://mlsanity.com/paper/2207.05705/) (2022)
- [Minimum norm interpolation by perceptra: Explicit regularization and implicit bias](https://mlsanity.com/paper/2311.06138/) (2023)
- [Efficient Learning of Long-Range and Equivariant Quantum Systems](https://mlsanity.com/paper/2312.17019/) (2023)
- [Generalization of Scaled Deep ResNets in the Mean-Field Regime](https://mlsanity.com/paper/2403.09889/) (2024)

## Abstract (arXiv)

We present a continuous formulation of machine learning, as a problem in the calculus of variations and differential-integral equations, in the spirit of classical numerical analysis. We demonstrate that conventional machine learning models and algorithms, such as the random feature model, the two-layer neural network model and the residual neural network model, can all be recovered (in a scaled form) as particular discretizations of different continuous formulations. We also present examples of new models, such as the flow-based random feature model, and new algorithms, such as the smoothed particle method and spectral method, that arise naturally from this continuous formulation. We discuss how the issues of generalization error and implicit regularization can be studied under this framework.