mlsanityData to 7 Oct 2026

FetchSGD: Communication-Efficient Federated Learning with Sketching

arXiv 2007.07682 ↗First posted 15 Jul 2020cs.LG stat.MLcited by 178

Not on the map. Why?

Abstract (arXiv)

Existing approaches to federated learning suffer from a communication bottleneck as well as convergence issues due to sparse client participation. In this paper we introduce a novel algorithm, called FetchSGD, to overcome these challenges. FetchSGD compresses model updates using a Count Sketch, and then takes advantage of the mergeability of sketches to combine model updates from many workers. A key insight in the design of FetchSGD is that, because the Count Sketch is linear, momentum and error accumulation can both be carried out within the sketch. This allows the algorithm to move momentum and error accumulation from clients to the central aggregator, overcoming the challenges of sparse client participation while still achieving high compression rates and good convergence. We prove that FetchSGD has favorable convergence guarantees, and we demonstrate its empirical effectiveness by training two residual networks and a transformer model.

Key facts. Cited by 178 papers in the map (data through 7 Oct 2026). First posted on arXiv 15 Jul 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 31; 2022: 40; 2023: 45; 2024: 38; 2025: 15; 2026: 9.

178 papers.

Sort:

By area:

  1. 2021A Field Guide to Federated Optimizationcited by 312
  2. 2021FedKD: Communication Efficient Federated Learning via Knowledge Distillationcited by 185
  3. 2021FedScale: Benchmarking Model and System Performance of Federated Learning at Scalecited by 135
  4. 2021Distributed Learning in Wireless Networks: Recent Progress and Future Challengescited by 125
  5. 2024Flora: Low-Rank Adapters Are Secretly Gradient Compressorscited by 103
  6. 2022Neurotoxin: Durable Backdoors in Federated Learningcited by 97
  7. 2024Selective Aggregation for Low-Rank Adaptation in Federated Learningcited by 80
  8. 2022NVIDIA FLARE: Federated Learning from Simulation to Real-Worldcited by 78
  9. 2022Personalized Federated Learning via Variational Bayesian Inferencecited by 68
  10. 2021From Distributed Machine Learning to Federated Learning: A Surveycited by 65
  11. 2021Provably Secure Federated Learning against Malicious Clientscited by 60
  12. 2023Federated Full-Parameter Tuning of Billion-Sized Language Models with Communication Cost under 18 Kilobytescited by 55
38 more
  1. 2021DRIVE: One-bit Distributed Mean Estimationcited by 52
  2. 2022Federated Learning via Decentralized Dataset Distillation in Resource-Constrained Edge Environmentscited by 49
  3. 2021SparseFed: Mitigating Model Poisoning Attacks in Federated Learning with Sparsificationcited by 46
  4. 2022Do Gradient Inversion Attacks Make Federated Learning Unsafe?cited by 44
  5. 2021Communication Efficiency in Federated Learning: Achievements and Challengescited by 40
  6. 2021DAdaQuant: Doubly-adaptive quantization for communication-efficient Federated Learningcited by 40
  7. 2022The Fundamental Price of Secure Aggregation in Differentially Private Federated Learningcited by 39
  8. 2023Efficient Personalized Federated Learning via Sparse Model-Adaptationcited by 38
  9. 2021Cross-Gradient Aggregation for Decentralized Learning from Non-IID datacited by 32
  10. 2021FedCV: A Federated Learning Framework for Diverse Computer Vision Taskscited by 32
  11. 2022Sparse Random Networks for Communication-Efficient Federated Learningcited by 32
  12. 2021DataLens: Scalable Privacy Preserving Training via Gradient Compression and Aggregationcited by 31
  13. 2021FedHM: Efficient Federated Learning for Heterogeneous Models via Low-rank Factorizationcited by 31
  14. 2022Acceleration of Federated Learning with Alleviated Forgetting in Local Trainingcited by 29
  15. 2022Fine-tuning Language Models over Slow Networks using Activation Compression with Guaranteescited by 29
  16. 2022A Multi-agent Reinforcement Learning Approach for Efficient Client Selection in Federated Learningcited by 28
  17. 2022FedSpace: An Efficient Federated Learning Framework at Satellites and Ground Stationscited by 28
  18. 2023DoCoFL: Downlink Compression for Cross-Device Federated Learningcited by 27
  19. 2022Sketching for First Order Method: Efficient Algorithm for Low-Bandwidth Channel and Vulnerabilitycited by 25
  20. 2022Accelerated Federated Learning with Decoupled Adaptive Optimizationcited by 23
  21. 2023GradientCoin: A Peer-to-Peer Decentralized Large Language Modelscited by 23
  22. 2021Learned Gradient Compression for Distributed Deep Learningcited by 20
  23. 2021EasyFL: A Low-code Federated Learning Platform For Dummiescited by 20
  24. 2022Scalable and Low-Latency Federated Learning with Cooperative Mobile Edge Networkingcited by 18
  25. 2024VeLoRA: Memory Efficient Training using Rank-1 Sub-Token Projectionscited by 18
  26. 2022FedOBD: Opportunistic Block Dropout for Efficiently Training Large-scale Neural Networks through Federated Learningcited by 17
  27. 2023A Survey for Federated Learning Evaluations: Goals and Measurescited by 16
  28. 2023On the Role of Server Momentum in Federated Learningcited by 16
  29. 2024From Challenges and Pitfalls to Recommendations and Opportunities: Implementing Federated Learning in Healthcarecited by 16
  30. 2025MuLoCo: Muon is a practical inner optimizer for DiLoCocited by 16
  31. 2022Decentral and Incentivized Federated Learning Frameworks: A Systematic Literature Reviewcited by 15
  32. 2021H-FL: A Hierarchical Communication-Efficient and Privacy-Protected Architecture for Federated Learningcited by 14
  33. 2022Optimizing the Communication-Accuracy Trade-off in Federated Learning with Rate-Distortion Theorycited by 14
  34. 2022Anchor Sampling for Federated Learning with Partial Client Participationcited by 14
  35. 2024Synergizing Foundation Models and Federated Learning: A Surveycited by 14
  36. 2022Federated Active Learning (F-AL): an Efficient Annotation Strategy for Federated Learningcited by 13
  37. 2022Federated Offline Reinforcement Learningcited by 13
  38. 2023MergeSFL: Split Federated Learning with Feature Merging and Batch Size Regulationcited by 13

50 most cited of 178 All 178 (JSON)