mlsanityData to 14 Oct 2026

Federated Averaging Langevin Dynamics: Toward a unified theory and new algorithms

arXiv 2211.00100 ↗First posted 31 Oct 2022stat.ML cs.LGcited by 2

Not on the map. Why?

Abstract (arXiv)

This paper focuses on Bayesian inference in a federated learning context (FL). While several distributed MCMC algorithms have been proposed, few consider the specific limitations of FL such as communication bottlenecks and statistical heterogeneity. Recently, Federated Averaging Langevin Dynamics (FALD) was introduced, which extends the Federated Averaging algorithm to Bayesian inference. We obtain a novel tight non-asymptotic upper bound on the Wasserstein distance to the global posterior for FALD. This bound highlights the effects of statistical heterogeneity, which causes a drift in the local updates that negatively impacts convergence. We propose a new algorithm VR-FALD* that uses control variates to correct the client drift. We establish non-asymptotic bounds showing that VR-FALD* is not affected by statistical heterogeneity. Finally, we illustrate our results on several FL benchmarks for Bayesian inference.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 42 of its 78 references that are in the map. First posted on arXiv 31 Oct 2022.

Builds on

42 of 78 references.

References · 42

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  3. 2017On Calibration of Modern Neural Networkscited by 4,960
  4. 2019Advances and Open Problems in Federated Learningcited by 3,046
  5. 2019Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shiftcited by 1,287
  6. 2019On the Convergence of FedAvg on Non-IID Datacited by 1,229
  7. 2020Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimizationcited by 906
  8. 2018signSGD: Compressed Optimisation for Non-Convex Problemscited by 571
  9. 2019A Simple Baseline for Bayesian Uncertainty in Deep Learningcited by 560
  10. 2019Federated Machine Learning: Concept and Applicationscited by 412
  11. 2019Bayesian Nonparametric Federated Learning of Neural Networkscited by 387
  12. 2018Sparsified SGD with Memorycited by 376
30 more
  1. 2021What Are Bayesian Neural Network Posteriors Really Like?cited by 308
  2. 2017Super-Convergence: Very Fast Training of Neural Networks Using Large Learning Ratescited by 199
  3. 2020FedBE: Making Bayesian Model Ensemble Applicable to Federated Learningcited by 176
  4. 2020Federated Learning with Compression: Unified Analysis and Sharp Guaranteescited by 160
  5. 2016QSGD: Communication-Efficient SGD via Gradient Quantization and Encodingcited by 147
  6. 2016High-dimensional Bayesian inference via the Unadjusted Langevin Algorithmcited by 146
  7. 2015Preconditioned Stochastic Gradient Langevin Dynamics for Deep Neural Networkscited by 121
  8. 2017User-friendly guarantees for the Langevin Monte Carlo with inaccurate gradientcited by 116
  9. 2020UVeQFed: Universal Vector Quantization for Federated Learningcited by 107
  10. 2020Local SGD: Unified Theory and New Efficient Methodscited by 88
  11. 20211-bit Adam: Communication Efficient Large-Scale Training with Adam's Convergence Speedcited by 75
  12. 2019Don't Jump Through Hoops and Remove Those Loops: SVRG and Katyusha are Better Without the Outer Loopcited by 68
  13. 2013Asymptotically Exact, Embarrassingly Parallel MCMCcited by 30
  14. 2021QLSD: Quantised Langevin stochastic dynamics for Bayesian federated learningcited by 26
  15. 2019Efficient MCMC Sampling with Dimension-Free Convergence Rate using ADMM-type Splittingcited by 22
  16. 2018On the Theory of Variance Reduction for Stochastic Gradient Monte Carlocited by 19
  17. 2020Federated Stochastic Gradient Langevin Dynamicscited by 16
  18. 2021On Convergence of Federated Averaging Langevin Dynamicscited by 14
  19. 2016Merging MCMC Subposteriors through Gaussian-Process Approximationscited by 13
  20. 2013Parallelizing MCMC via Weierstrass Samplercited by 11
  21. 2019Embarrassingly parallel MCMC using deep invertible transformationscited by 11
  22. 2021DG-LMC: A Turn-key and Scalable Synchronous Distributed MCMC Algorithm via Langevin Monte Carlo within Gibbscited by 11
  23. 2022Parallel MCMC Without Embarrassing Failurescited by 11
  24. 2015Parallelizing MCMC with Random Partition Treescited by 10
  25. 2021Communication-Efficient Agnostic Federated Averagingcited by 10
  26. 2019Asymptotically exact data augmentation: models, properties and algorithmscited by 8
  27. 2014Distributed Estimation, Information Loss and Exponential Familiescited by 6
  28. 2021Bayesian Fusion: Scalable unification of distributed statistical analysescited by 5
  29. 2022Federated Learning with a Sampling Algorithm under Isoperimetrycited by 4
  30. 2022Leveraging Channel Noise for Sampling and Privacy via Quantized Federated Langevin Monte Carlocited by 3

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 1.

2 papers.

  1. 2023ELF: Federated Langevin Algorithms with Primal, Dual and Bidirectional Compressioncited by 4
  2. 2024Bayesian Federated Learning with Hamiltonian Monte Carlo: Algorithm and Theorycited by 4