mlsanityData to 14 Oct 2026

On Convergence of Federated Averaging Langevin Dynamics

arXiv 2112.05120 ↗First posted 9 Dec 2021stat.ML cs.LGcited by 14

Not on the map. Why?

Abstract (arXiv)

We propose a federated averaging Langevin algorithm (FA-LD) for uncertainty quantification and mean predictions with distributed clients. In particular, we generalize beyond normal posterior distributions and consider a general class of models. We develop theoretical guarantees for FA-LD for strongly log-concave distributions with non-i.i.d data and study how the injected noise and the stochastic-gradient noise, the heterogeneity of data, and the varying learning rates affect the convergence. Such an analysis sheds light on the optimal choice of local updates to minimize communication costs. Important to our approach is that the communication efficiency does not deteriorate with the injected noise in the Langevin algorithms. In addition, we examine in our FA-LD algorithm both independent and correlated noise used over different clients. We observe there is a trade-off between the pairs among communication, accuracy, and data privacy. As local devices may become inactive in federated networks, we also show convergence results based on different averaging schemes where only partial device updates are available. In such a case, we discover an additional bias that does not decay to zero.

Key facts. Cited by 14 papers in the map (data through 14 Oct 2026). Builds on 55 of its 93 references that are in the map. First posted on arXiv 9 Dec 2021.

Builds on

55 of 93 references.

References · 55

  1. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  2. 2017On Calibration of Modern Neural Networkscited by 4,960
  3. 2018Federated Optimization in Heterogeneous Networkscited by 2,987
  4. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  5. 2019Federated Learning: Challenges, Methods, and Future Directionscited by 1,538
  6. 2019On the Convergence of FedAvg on Non-IID Datacited by 1,229
  7. 2018Local SGD Converges Fast and Communicates Littlecited by 605
  8. 2021FedBN: Federated Learning on Non-IID Features via Local Batch Normalizationcited by 571
  9. 2019Federated Learning with Differential Privacy: Algorithms and Performance Analysiscited by 536
  10. 2018Adaptive Federated Learning in Resource Constrained Edge Computing Systemscited by 409
  11. 2014Stochastic Gradient Hamiltonian Monte Carlocited by 343
  12. 2018Parallel Restarted SGD with Faster Convergence and Less Communication: Demystifying Why Model Averaging Works for Deep Learningcited by 278
38 more
  1. 2019A Joint Learning and Communications Framework for Federated Learning over Wireless Networkscited by 263
  2. 2017Non-convex learning via Stochastic Gradient Langevin Dynamics: a nonasymptotic analysiscited by 243
  3. 2018Privacy Amplification by Subsampling: Tight Analyses via Couplings and Divergencescited by 242
  4. 2019Decentralized Stochastic Optimization and Gossip Algorithms with Compressed Communicationcited by 221
  5. 2014Theoretical guarantees for approximate sampling from smooth and log-concave densitiescited by 208
  6. 2020FedBE: Making Bayesian Model Ensemble Applicable to Federated Learningcited by 176
  7. 2019Privacy-preserving Federated Brain Tumour Segmentationcited by 169
  8. 2019On the Convergence of Local Descent Methods in Federated Learningcited by 153
  9. 2020Minibatch vs Local SGD for Heterogeneous Distributed Learningcited by 147
  10. 2015A Complete Recipe for Stochastic Gradient MCMCcited by 146
  11. 2016High-dimensional Bayesian inference via the Unadjusted Langevin Algorithmcited by 146
  12. 2017Underdamped Langevin MCMC: A non-asymptotic analysiscited by 133
  13. 2018Analysis of Langevin Monte Carlo via convex optimizationcited by 127
  14. 2020Multi-site fMRI Analysis Using Privacy-preserving Federated Learning and Domain Adaptation: ABIDE Resultscited by 122
  15. 2017User-friendly guarantees for the Langevin Monte Carlo with inaccurate gradientcited by 116
  16. 2020FedSplit: An algorithmic framework for fast federated optimizationcited by 108
  17. 2019First Analysis of Local GD on Heterogeneous Datacited by 81
  18. 2019The Randomized Midpoint Method for Log-Concave Samplingcited by 78
  19. 2020Federated Learning via Posterior Averaging: A New Perspective and Practical Algorithmscited by 78
  20. 2017A Hitting Time Analysis of Stochastic Gradient Langevin Dynamicscited by 71
  21. 2018Sampling Can Be Faster Than Optimizationcited by 68
  22. 2014Consistency and fluctuations for stochastic gradient Langevin dynamicscited by 65
  23. 2015Privacy for Free: Posterior Sampling and Stochastic Gradient Monte Carlocited by 63
  24. 2019High-Order Langevin Diffusion Yields an Accelerated MCMC Algorithmcited by 36
  25. 2020On the Outsized Importance of Learning Rates in Local Update Methodscited by 36
  26. 2016On the Convergence of Stochastic Gradient MCMC Algorithms with High-Order Integratorscited by 35
  27. 2020Non-convex Learning via Replica Exchange Stochastic Gradient MCMCcited by 34
  28. 2019Stochastic Runge-Kutta Accelerates Langevin Monte Carlo and Beyondcited by 33
  29. 2020TextHide: Tackling Data Privacy in Language Understanding Taskscited by 33
  30. 2013Asymptotically Exact, Embarrassingly Parallel MCMCcited by 30
  31. 2019Optimal Convergence Rate of Hamiltonian Monte Carlo for Strongly Logconcave Distributionscited by 28
  32. 2020A Contour Stochastic Gradient Langevin Dynamics Algorithm for Simulations of Multi-modal Distributionscited by 26
  33. 2021QLSD: Quantised Langevin stochastic dynamics for Bayesian federated learningcited by 26
  34. 2021Sqrt(d) Dimension Dependence of Langevin Monte Carlocited by 25
  35. 2019Is There an Analog of Nesterov Acceleration for MCMC?cited by 19
  36. 2020Decentralized Stochastic Gradient Langevin Dynamics and Hamiltonian Monte Carlocited by 19
  37. 2020Federated Stochastic Gradient Langevin Dynamicscited by 16
  38. 2019"The Squawk Bot": Joint Learning of Time Series and Text Data Modalities for Automated Financial Information Filteringcited by 13

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year

Citing papers per year: 2022: 5; 2023: 4; 2024: 3; 2025: 2.

14 papers.

Sort:

  1. 2023Towards Building the Federated GPT: Federated Instruction Tuningcited by 185
  2. 2022FedPop: A Bayesian Approach for Personalised Federated Learningcited by 35
  3. 2023D2NO: Efficient Handling of Heterogeneous Input Function Spaces with Distributed Deep Neural Operatorscited by 18
  4. 2022Interacting Contour Stochastic Gradient Langevin Dynamicscited by 9
  5. 2022Federated Learning with a Sampling Algorithm under Isoperimetrycited by 4
  6. 2022A Convergence Theory for Federated Average: Beyond Smoothnesscited by 4
  7. 2023ELF: Federated Langevin Algorithms with Primal, Dual and Bidirectional Compressioncited by 4
  8. 2024Bayesian Federated Learning with Hamiltonian Monte Carlo: Algorithm and Theorycited by 4
  9. 2025Connecting Federated ADMM to Bayescited by 4
  10. 2023DP-Fast MH: Private, Fast, and Accurate Metropolis-Hastings for Large-Scale Bayesian Inferencecited by 3
  11. 2022Federated Averaging Langevin Dynamics: Toward a unified theory and new algorithmscited by 2
  12. 2024Trustworthy Personalized Bayesian Federated Learning via Posterior Fine-Tunecited by 1
2 more
  1. 2024Distributed Markov Chain Monte Carlo Sampling based on the Alternating Direction Method of Multiplierscited by 0
  2. 2025Bayesian Federated Learning for Continual Trainingcited by 0