mlsanityData to 7 Oct 2026

Byzantine Machine Learning Made Easy by Resilient Averaging of Momentums

arXiv 2205.12173 ↗First posted 24 May 2022cs.LG cs.DCcited by 63

Not on the map. Why?

Abstract (arXiv)

Byzantine resilience emerged as a prominent topic within the distributed machine learning community. Essentially, the goal is to enhance distributed optimization algorithms, such as distributed SGD, in a way that guarantees convergence despite the presence of some misbehaving (a.k.a., em Byzantine) workers. Although a myriad of techniques addressing the problem have been proposed, the field arguably rests on fragile foundations. These techniques are hard to prove correct and rely on assumptions that are (a) quite unrealistic, i.e., often violated in practice, and (b) heterogeneous, i.e., making it difficult to compare approaches. We present RESAM (RESilient Averaging of Momentums), a unified framework that makes it simple to establish optimal Byzantine resilience, relying only on standard machine learning assumptions. Our framework is mainly composed of two operators: resilient averaging at the server and distributed momentum at the workers. We prove a general theorem stating the convergence of distributed SGD under RESAM. Interestingly, demonstrating and comparing the convergence of many existing techniques become direct corollaries of our theorem, without resorting to stringent assumptions. We also present an empirical evaluation of the practical relevance of RESAM.

Key facts. Cited by 63 papers in the map (data through 7 Oct 2026). Builds on 41 of its 66 references that are in the map. First posted on arXiv 24 May 2022.

Builds on

41 of 66 references.

References · 41

  1. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,283
  2. 2019Advances and Open Problems in Federated Learningcited by 3,021
  3. 2021Convex optimizationcited by 2,479
  4. 2016Optimization Methods for Large-Scale Machine Learningcited by 1,140
  5. 2013Stochastic First- and Zeroth-order Methods for Nonconvex Stochastic Programmingcited by 716
  6. 2018Byzantine-Robust Distributed Learning: Towards Optimal Statistical Ratescited by 698
  7. 2018The Hidden Vulnerability of Distributed Learning in Byzantiumcited by 324
  8. 2019Robust Aggregation for Federated Learningcited by 314
  9. 2019Momentum-Based Variance Reduction in Non-Convex SGDcited by 302
  10. 2019A Little Is Enough: Circumventing Defenses For Distributed Learningcited by 285
  11. 2016Robust Estimators in High Dimensions without the Computational Intractabilitycited by 209
  12. 2018RSA: Byzantine-Robust Stochastic Aggregation Methods for Distributed Learning from Heterogeneous Datasetscited by 148
29 more
  1. 2020Learning from History for Byzantine Robust Optimizationcited by 137
  2. 2019Fall of Empires: Breaking Byzantine-tolerant SGD by Inner Product Manipulationcited by 136
  3. 2016Agnostic Estimation of Mean and Covariancecited by 131
  4. 2020Byzantine-Robust Learning on Heterogeneous Datasets via Bucketingcited by 125
  5. 2018Sever: A Robust Meta-Algorithm for Stochastic Optimizationcited by 123
  6. 2015Asynchronous Parallel Stochastic Gradient for Nonconvex Optimizationcited by 115
  7. 2016Statistical Query Lower Bounds for Robust Estimation of High-dimensional Gaussians and Gaussian Mixturescited by 114
  8. 2018Generalized Byzantine-tolerant SGDcited by 94
  9. 2016Learning from Untrusted Datacited by 90
  10. 2018Byzantine Stochastic Gradient Descentcited by 81
  11. 2018Robust Estimation via Robust Gradient Estimationcited by 76
  12. 2018DRACO: Byzantine-resilient Distributed Training via Redundant Gradientscited by 74
  13. 2017Distributed Statistical Machine Learning in Adversarial Settings: Byzantine Gradient Descentcited by 59
  14. 2020Byzantine-Resilient Non-Convex Stochastic Gradient Descentcited by 56
  15. 2020Collaborative Learning in the Jungle (Decentralized, Byzantine, Heterogeneous, Asynchronous and Nonconvex Learning)cited by 54
  16. 2019Zeno++: Robust Fully Asynchronous SGDcited by 53
  17. 2019Stochastic Gradient Descent for Nonconvex Learning without Bounded Gradient Assumptionscited by 48
  18. 2019DETOX: A Redundancy-based Framework for Faster and More Robust Gradient Aggregationcited by 44
  19. 2017Resilience: A Criterion for Learning in the Presence of Arbitrary Outlierscited by 43
  20. 2016Geometric Median in Nearly Linear Timecited by 38
  21. 2018Defending Against Saddle Point Attack in Byzantine-Robust Distributed Learningcited by 32
  22. 2020Byzantine-Resilient High-Dimensional Federated Learningcited by 27
  23. 2021Approximate Byzantine Fault-Tolerance in Distributed Optimizationcited by 27
  24. 2019Federated Learning with Unbiased Gradient Aggregation and Controllable Meta Updatingcited by 22
  25. 2019Data Encoding for Byzantine-Resilient Distributed Optimizationcited by 10
  26. 2021On the Strategyproofness of the Geometric Mediancited by 9
  27. 2018Finite-time Guarantees for Byzantine-Resilient Distributed State Estimation with Noisy Measurementscited by 4
  28. 2017Outlier Robust Online Learningcited by 2
  29. 2019Randomized Reactive Redundancy for Byzantine Fault-Tolerance in Parallelized Learningcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 3; 2023: 16; 2024: 20; 2025: 17; 2026: 7.

63 papers.

Sort:

By area:

  1. 2023Fixing by Mixing: A Recipe for Optimal Byzantine ML under Heterogeneitycited by 64
  2. 2022On the Impossible Safety of Large AI Modelscited by 27
  3. 2023Robust Distributed Learning: Tight Error Bounds and Breakdown Point under Data Heterogeneitycited by 25
  4. 2023On the Privacy-Robustness-Utility Trilemma in Distributed Learningcited by 24
  5. 2023Byzantine-Robust Learning on Heterogeneous Data via Gradient Splittingcited by 13
  6. 2024Byzantine-Robust Federated Learning: Impact of Client Subsampling and Local Updatescited by 13
  7. 2024Achieving Byzantine-Resilient Federated Learning via Layer-Adaptive Sparsified Model Aggregationcited by 13
  8. 2025Detecting Backdoor Attacks in Federated Learning via Direction Alignment Inspectioncited by 13
  9. 2024MedBN: Robust Test-Time Adaptation against Malicious Test Samplescited by 10
  10. 2024Adaptive Gradient Clipping for Robust Federated Learningcited by 10
  11. 2022Byzantines can also Learn from History: Fall of Centered Clipping in Federated Learningcited by 8
  12. 2023Byzantine-Robust Distributed Online Learning: Taming Adversarial Participants in An Adversarial Environmentcited by 8
38 more
  1. 2023On the Tradeoff between Privacy Preservation and Byzantine-Robustness in Decentralized Learningcited by 8
  2. 2024Unified Breakdown Analysis for Byzantine Robust Gossipcited by 8
  3. 2025Towards Trustworthy Federated Learning with Untrusted Participantscited by 8
  4. 2023Byzantine-Resilient Federated Learning at Edgecited by 6
  5. 2024Global Convergence Guarantees for Federated Policy Gradient Methods with Adversariescited by 6
  6. 2023Federated Learning in the Presence of Adversarial Client Unavailabilitycited by 5
  7. 2024Dynamic Byzantine-Robust Learning: Adapting to Switching Byzantine Workerscited by 5
  8. 2023Near-Optimal Resilient Aggregation Rules for Distributed Learning Using 1-Center and 1-Mean Clustering with Outlierscited by 4
  9. 2024A survey on secure decentralized optimization and learningcited by 4
  10. 2024Overcoming the Challenges of Batch Normalization in Federated Learningcited by 3
  11. 2024Fault Tolerant ML: Efficient Meta-Aggregation and Synchronous Trainingcited by 3
  12. 2024Private Geometric Mediancited by 3
  13. 2024Review of Mathematical Optimization in Federated Learningcited by 3
  14. 2025ByzFL: Research Framework for Robust Federated Learningcited by 3
  15. 2025Delayed Momentum Aggregation: Communication-efficient Byzantine-robust Federated Learning with Partial Participationcited by 3
  16. 2023Survey of Federated Learning Models for Spatial-Temporal Mobility Applicationscited by 2
  17. 2024Decentralized Federated Policy Gradient with Byzantine Fault-Tolerance and Provably Fast Convergencecited by 2
  18. 2024TernaryVote: Differentially Private, Communication Efficient, and Byzantine Resilient Distributed Optimization on Heterogeneous Datacited by 2
  19. 2024On the Relevance of Byzantine Robust Optimization Against Data Poisoningcited by 2
  20. 2025Weight for Robustness: A Comprehensive Approach towards Optimal Fault-Tolerant Asynchronous MLcited by 2
  21. 2025Tight Stability Bounds for Robust Distributed Learning: Byzantine Failures Hurt Generalization More than Data Poisoningcited by 2
  22. 2025Robust Federated Inferencecited by 2
  23. 2022FedCut: A Spectral Analysis Framework for Reliable Detection of Byzantine Colluderscited by 1
  24. 2023Exploring the Robustness of Decentralized Training for Large Language Modelscited by 1
  25. 2023Balancing Privacy, Robustness, and Efficiency in Machine Learningcited by 1
  26. 2024Byzantine-Robust Gossip: Insights from a Dual Approachcited by 1
  27. 2024Understanding Byzantine Robustness in Federated Learning with A Black-box Servercited by 1
  28. 2024Byzantine-Robust and Communication-Efficient Distributed Learning via Compressed Momentum Filteringcited by 1
  29. 2025Reconciling Communication Compression and Byzantine-Robustness in Distributed Learningcited by 1
  30. 2025A Nesterov-Accelerated Byzantine-Robust Federated Learningcited by 1
  31. 2025FLARE: Adaptive Multi-Dimensional Reputation for Robust Client Reliability in Federated Learningcited by 1
  32. 2023Flag Aggregator: Scalable Distributed Training under Failures and Augmented Losses using Convex Optimizationcited by 0
  33. 2023On the Optimal Batch Size for Byzantine-Robust Distributed Learningcited by 0
  34. 2023SABLE: Secure And Byzantine robust LEarningcited by 0
  35. 2023Identifying the Truth of Global Model: A Generic Solution to Defend Against Byzantine and Backdoor Attacks in Federated Learning (full version)cited by 0
  36. 2024Aggressive or Imperceptible, or Both: Network Pruning Assisted Hybrid Byzantines in Federated Learningcited by 0
  37. 2024LiD-FL: Towards List-Decodable Federated Learningcited by 0
  38. 2025FedTilt: Towards Multi-Level Fairness-Preserving and Robust Federated Learningcited by 0

50 most cited of 63 All 63 (JSON)