mlsanityData to 7 Oct 2026

Propagation of Chaos for Mean-Field Langevin Dynamics and its Application to Model Ensemble

arXiv 2502.05784 ↗First posted 9 Feb 2025stat.ML cs.LGcited by 5

Not on the map. Why?

Abstract (arXiv)

Mean-field Langevin dynamics (MFLD) is an optimization method derived by taking the mean-field limit of noisy gradient descent for two-layer neural networks in the mean-field regime. Recently, the propagation of chaos (PoC) for MFLD has gained attention as it provides a quantitative characterization of the optimization complexity in terms of the number of particles and iterations. A remarkable progress by Chen et al. (2022) showed that the approximation error due to finite particles remains uniform in time and diminishes as the number of particles increases. In this paper, by refining the defective log-Sobolev inequality -- a key result from that earlier work -- under the neural network training setting, we establish an improved PoC result for MFLD, which removes the exponential dependence on the regularization coefficient from the particle approximation term of the optimization complexity. As an application, we propose a PoC-based model ensemble strategy with theoretical guarantees.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 37 of its 59 references that are in the map. First posted on arXiv 9 Feb 2025.

Builds on

37 of 59 references.

References · 37

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
  4. 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,194
  5. 2023Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunchcited by 679
  6. 2019Linear Mode Connectivity and the Lottery Ticket Hypothesiscited by 627
  7. 2015Train faster, generalize better: Stability of stochastic gradient descentcited by 547
  8. 2018A Mean Field View of the Landscape of Two-Layers Neural Networkscited by 463
  9. 2018On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transportcited by 422
  10. 2022Dataless Knowledge Fusion by Merging Weights of Language Modelscited by 384
  11. 2020What is being transferred in transfer learning?cited by 365
  12. 2023LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Modelscited by 317
25 more
  1. 2021Ensemble deep learning: A reviewcited by 283
  2. 2024Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunitiescited by 247
  3. 2019Rapid Convergence of the Unadjusted Langevin Algorithm: Isoperimetry Sufficescited by 224
  4. 2023Task Arithmetic in the Tangent Space: Improved Editing of Pre-Trained Modelscited by 218
  5. 2024Arcee's MergeKit: A Toolkit for Merging Large Language Modelscited by 189
  6. 2022Patching open-vocabulary models by interpolating weightscited by 187
  7. 2020Feature Learning in Infinite-Width Neural Networkscited by 182
  8. 2022High-dimensional Asymptotics of Feature Learning: How One Gradient Step Improves the Representationcited by 146
  9. 2023Model Breadcrumbs: Scaling Multi-Task Model Merging with Sparse Maskscited by 130
  10. 2022Model Ratatouille: Recycling Diverse Models for Out-of-Distribution Generalizationcited by 97
  11. 2023AdapterSoup: Weight Averaging to Improve Generalization of Pretrained Language Modelscited by 84
  12. 2022Mean-Field Langevin Dynamics: Exponential Convergence and Annealingcited by 60
  13. 2022Convex Analysis of the Mean Field Langevin Dynamicscited by 53
  14. 2017Stochastic Particle Gradient Descent for Infinite Ensemblescited by 52
  15. 2017The Relative Performance of Ensemble Methods with Deep Convolutional Neural Networks for Image Classificationcited by 50
  16. 2019Mean-Field Langevin Dynamics and Energy Landscape of Neural Networkscited by 49
  17. 2022Uniform-in-time propagation of chaos for mean field Langevin dynamicscited by 23
  18. 2019Global convergence of neuron birth-death dynamicscited by 22
  19. 2023Convergence of mean-field Langevin dynamics: Time and space discretization, stochastic gradient, and variance reductioncited by 19
  20. 2019Mean-Field Neural ODEs via Relaxed Optimal Controlcited by 16
  21. 2024Learning Multi-Index Models with Neural Networks via Mean-Field Langevin Dynamicscited by 14
  22. 2022Entropic Fictitious Play for Mean Field Optimization Problemcited by 12
  23. 2024Improved Particle Approximation Error for Mean Field Neural Networkscited by 12
  24. 2024Automatic Detection of Depression in Speech Using Ensemble Convolutional Neural Networkscited by 7
  25. 2024Merging in a Bottle: Differentiable Adaptive Merging (DAM) and the Path from Averaging to Automationcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 4.

5 papers.

  1. 2025Mirror Mean-Field Langevin Dynamicscited by 3
  2. 2026Weight-Decay Turns Transformer Loss Landscapes Villani: Functional-Analytic Foundations for Optimization and Generalizationcited by 1
  3. 2026Thinned Mean Field Langevin Dynamicscited by 1
  4. 2026On propagation of chaos for the Fisher-Rao gradient flow in entropic mean-field optimizationcited by 0
  5. 2026A Distributional Optimisation Perspective on Combining Models in Deep Learningcited by 0