# Global convergence of neuron birth-death dynamics

arXiv: https://arxiv.org/abs/1902.01843 · HTML: https://mlsanity.com/paper/1902.01843/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Feb 2019.

## Built on by (most cited first, 22 of 22)

- [Neural Galerkin Schemes with Active Learning for High-Dimensional Evolution Equations](https://mlsanity.com/paper/2203.01360/) (2022)
- [Convex Analysis of the Mean Field Langevin Dynamics](https://mlsanity.com/paper/2201.10469/) (2022)
- [Unbalanced Optimal Transport, from Theory to Numerics](https://mlsanity.com/paper/2211.08775/) (2022)
- [Transformers Learn Nonlinear Features In Context: Nonconvex Mean-field Dynamics on the Attention Landscape](https://mlsanity.com/paper/2402.01258/) (2024)
- [Birth-death dynamics for sampling: Global convergence, approximations and their asymptotics](https://mlsanity.com/paper/2211.00450/) (2022)
- [An Explicit Expansion of the Kullback-Leibler Divergence along its Fisher-Rao Gradient Flow](https://mlsanity.com/paper/2302.12229/) (2023)
- [Analysis of feature learning in weight-tied autoencoders via the mean field lens](https://mlsanity.com/paper/2102.08373/) (2021)
- [Two-Scale Gradient Descent Ascent Dynamics Finds Mixed Nash Equilibria of Continuous Games: A Mean-Field Perspective](https://mlsanity.com/paper/2212.08791/) (2022)
- [Fisher-Rao Gradient Flow: Geodesic Convexity and Functional Inequalities](https://mlsanity.com/paper/2407.15693/) (2024)
- [Weighted quantization using MMD: From mean field to mean shift via gradient flows](https://mlsanity.com/paper/2502.10600/) (2025)
- [Parallel Deep Neural Networks Have Zero Duality Gap](https://mlsanity.com/paper/2110.06482/) (2021)
- [Provably convergent quasistatic dynamics for mean-field two-player zero-sum games](https://mlsanity.com/paper/2202.10947/) (2022)
- [GAD-PVI: A General Accelerated Dynamic-Weight Particle-Based Variational Inference Framework](https://mlsanity.com/paper/2312.16429/) (2023)
- [Dual Training of Energy-Based Models with Overparametrized Shallow Neural Networks](https://mlsanity.com/paper/2107.05134/) (2021)
- [DPVI: A Dynamic-Weight Particle-Based Variational Inference Framework](https://mlsanity.com/paper/2112.00945/) (2021)
- [Limiting fluctuation and trajectorial stability of multilayer neural networks with mean field training](https://mlsanity.com/paper/2110.15954/) (2021)
- [Variational Inference of overparameterized Bayesian Neural Networks: a theoretical and empirical study](https://mlsanity.com/paper/2207.03859/) (2022)
- [Propagation of Chaos for Mean-Field Langevin Dynamics and its Application to Model Ensemble](https://mlsanity.com/paper/2502.05784/) (2025)
- [Ultra-fast feature learning for the training of two-layer neural networks in the two-timescale regime](https://mlsanity.com/paper/2504.18208/) (2025)
- [Mean Field Optimization Problem Regularized by Fisher Information](https://mlsanity.com/paper/2302.05938/) (2023)
- [Leveraging Optimal Transport via Projections on Subspaces for Machine Learning Applications](https://mlsanity.com/paper/2311.13883/) (2023)
- [Towards Optimal Randomized Strategies in Adversarial Example Game](https://mlsanity.com/paper/2306.16738/) (2023)

## Abstract (arXiv)

Neural networks with a large number of parameters admit a mean-field description, which has recently served as a theoretical explanation for the favorable training properties of "overparameterized" models. In this regime, gradient descent obeys a deterministic partial differential equation (PDE) that converges to a globally optimal solution for networks with a single hidden layer under appropriate assumptions. In this work, we propose a non-local mass transport dynamics that leads to a modified PDE with the same minimizer. We implement this non-local dynamics as a stochastic neuronal birth-death process and we prove that it accelerates the rate of convergence in the mean-field limit. We subsequently realize this PDE with two classes of numerical schemes that converge to the mean-field equation, each of which can easily be implemented for neural networks with finite numbers of parameters. We illustrate our algorithms with two models to provide intuition for the mechanism through which convergence is accelerated.