mlsanityData to 7 Oct 2026

Communication-Efficient Byzantine-Resilient Federated Zero-Order Optimization

arXiv 2406.14362 ↗First posted 20 Jun 2024cs.LG cs.AIcited by 4

Not on the map. Why?

Abstract (arXiv)

We introduce CYBER-0, the first zero-order optimization algorithm for memory-and-communication efficient Federated Learning, resilient to Byzantine faults. We show through extensive numerical experiments on the MNIST dataset and finetuning RoBERTa-Large that CYBER-0 outperforms state-of-the-art algorithms in terms of communication and memory efficiency while reaching similar accuracy. We provide theoretical guarantees on its convergence for convex loss functions.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 36 of its 50 references that are in the map. First posted on arXiv 20 Jun 2024.

Builds on

36 of 50 references.

References · 36

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  3. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  4. 2015A large annotated corpus for learning natural language inferencecited by 1,837
  5. 2018Federated Learning with Non-IID Datacited by 1,099
  6. 2018Byzantine-Robust Distributed Learning: Towards Optimal Statistical Ratescited by 698
  7. 2019Square Attack: a query-efficient black-box adversarial attack via random searchcited by 682
  8. 2018signSGD: Compressed Optimisation for Non-Convex Problemscited by 561
  9. 2017Evolution Strategies as a Scalable Alternative to Reinforcement Learningcited by 555
  10. 2019Federated Learning with Differential Privacy: Algorithms and Performance Analysiscited by 528
  11. 2018Black-box Adversarial Attacks with Limited Queries and Informationcited by 499
  12. 2019Local Model Poisoning Attacks to Byzantine-Robust Federated Learningcited by 479
24 more
  1. 2021Federated Learning on Non-IID Data: A Surveycited by 345
  2. 2023Fine-Tuning Language Models with Just Forward Passescited by 337
  3. 2019Differential Privacy Has Disparate Impact on Model Accuracycited by 316
  4. 2019Error Feedback Fixes SignSGD and other Gradient Compression Schemescited by 306
  5. 2020Data Poisoning Attacks Against Federated Learning Systemscited by 266
  6. 2017TernGrad: Ternary Gradients to Reduce Communication in Distributed Deep Learningcited by 253
  7. 2019PowerSGD: Practical Low-Rank Gradient Compression for Distributed Optimizationcited by 234
  8. 2015Coordinate Descent Algorithmscited by 218
  9. 2018A Hybrid Approach to Privacy-Preserving Federated Learningcited by 208
  10. 2018Mitigating Sybils in Federated Learning Poisoningcited by 158
  11. 2020Learning to Detect Malicious Clients for Robust Federated Learningcited by 86
  12. 2022Communication-Efficient Stochastic Zeroth-Order Optimization for Federated Learningcited by 59
  13. 2023Federated Full-Parameter Tuning of Billion-Sized Language Models with Communication Cost under 18 Kilobytescited by 55
  14. 2019Wasserstein Robust Reinforcement Learningcited by 50
  15. 2024Private Fine-tuning of Large Language Models with Zeroth-order Optimizationcited by 41
  16. 2021An Efficient Statistical-based Gradient Compression Technique for Distributed Training Systemscited by 32
  17. 2021Byzantine-robust Federated Learning through Collaborative Malicious Gradient Filteringcited by 28
  18. 2021CodedPaddedFL and CodedSecAgg: Straggler Mitigation and Secure Aggregation in Federated Learningcited by 20
  19. 2022SwiftAgg+: Achieving Asymptotically Optimal Communication Loads in Secure Aggregation for Federated Learningcited by 18
  20. 2023Just One Byte (per gradient): A Note on Low-Bandwidth Decentralized Language Model Finetuning Using Shared Randomnesscited by 18
  21. 2023z-SignFedAvg: A Unified Stochastic Sign-based Compression for Federated Learningcited by 17
  22. 2021Basil: A Fast and Byzantine-Resilient Approach for Decentralized Trainingcited by 13
  23. 2024Byzantine-Resilient Secure Aggregation for Federated Learning Without Privacy Compromisescited by 7
  24. 2023LASER: Linear Compression in Wireless Distributed Optimizationcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 1; 2026: 2.

4 papers.

  1. 2024Fed-ZOE: Communication-Efficient Over-the-Air Federated Learning via Zeroth-Order Estimationcited by 1
  2. 2025Private Aggregation for Byzantine-Resilient Heterogeneous Federated Learningcited by 0
  3. 2026Adversary-Robust Learning from Fully Asynchronous Directional Derivative Estimatescited by 0
  4. 2026AirGC-CD: Gaussian-Circulant Precoding for Exactly Debiasable PAPR Reduction in Over-the-Air Federated Learningcited by 0