mlsanityData to 7 Oct 2026

Byzantine Robustness and Partial Participation Can Be Achieved at Once: Just Clip Gradient Differences

arXiv 2311.14127 ↗First posted 23 Nov 2023cs.LG cs.AI cs.DCcited by 5

Not on the map. Why?

Abstract (arXiv)

Distributed learning has emerged as a leading paradigm for training large machine learning models. However, in real-world scenarios, participants may be unreliable or malicious, posing a significant challenge to the integrity and accuracy of the trained models. Byzantine fault tolerance mechanisms have been proposed to address these issues, but they often assume full participation from all clients, which is not always practical due to the unavailability of some clients or communication constraints. In our work, we propose the first distributed method with client sampling and provable tolerance to Byzantine workers. The key idea behind the developed method is the use of gradient clipping to control stochastic gradient differences in recursive variance reduction. This allows us to bound the potential harm caused by Byzantine workers, even during iterations when all sampled clients are Byzantine. Furthermore, we incorporate communication compression into the method to enhance communication efficiency. Under general assumptions, we prove convergence rates for the proposed method that match the existing state-of-the-art (SOTA) theoretical results. We also propose a heuristic on adjusting any Byzantine-robust method to a partial participation scenario via clipping.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 19 of its 90 references that are in the map. First posted on arXiv 23 Nov 2023.

Builds on

19 of 90 references.

References · 19

  1. 2019Why gradient clipping accelerates training: A theoretical justification for adaptivitycited by 335
  2. 2020Privacy and Robustness in Federated Learning: Attacks and Defensescited by 163
  3. 2020On Biased Compression for Distributed Learningcited by 140
  4. 2019Distributed Learning with Compressed Gradient Differencescited by 110
  5. 2020On the Convergence of SGD with Biased Gradientscited by 79
  6. 2019Stochastic Distributed Learning with Gradient Quantization and Variance Reductioncited by 64
  7. 2023High-Probability Bounds for Stochastic Optimization and Variational Inequalities: the Case of Unbounded Variancecited by 60
  8. 2018Distributed learning with compressed gradientscited by 53
  9. 2021FedNL: Making Newton-Type Methods Applicable to Federated Learningcited by 50
  10. 2020A Reputation Mechanism Is All You Need: Collaborative Fairness and Adversarial Robustness in Federated Learningcited by 38
  11. 2022Byzantine-Robust Decentralized Learning via ClippedGossipcited by 29
  12. 2022Federated Optimization Algorithms with Random Reshuffling and Gradient Compressioncited by 18
7 more
  1. 2021BROADCAST: Reducing Both Stochastic and Compression Noise to Robustify Communication-Efficient Federated Learningcited by 12
  2. 2021Secure Distributed Training at Scalecited by 11
  3. 2023Improved Convergence in High Probability of Clipped Gradient Methods with Heavy Tailscited by 11
  4. 2021Distributed Deep Learning Using Volunteer Computing-Like Paradigmcited by 8
  5. 2020Dynamic Defense Against Byzantine Poisoning Attacks in Federated Learningcited by 6
  6. 2021EF21 with Bells & Whistles: Six Algorithmic Extensions of Modern Error Feedbackcited by 6
  7. 2020ByGARS: Byzantine SGD with Arbitrary Number of Attackerscited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 3.

5 papers.

  1. 2026Enhancing LLM Training via Spectral Clippingcited by 5
  2. 2025Incentivizing Permissionless Distributed Learning of LLMscited by 3
  3. 2025Delayed Momentum Aggregation: Communication-efficient Byzantine-robust Federated Learning with Partial Participationcited by 3
  4. 2026From Inexact Gradients to Byzantine Robustness: Acceleration and Optimization under Similaritycited by 0
  5. 2026Robustifying Asynchronous SGD via Soft Throttlingcited by 0