mlsanityData to 14 Oct 2026

Optimizing the Communication-Accuracy Trade-off in Federated Learning with Rate-Distortion Theory

arXiv 2201.02664 ↗First posted 7 Jan 2022cs.LG cs.DC cs.ITcited by 14

Not on the map. Why?

Abstract (arXiv)

A significant bottleneck in federated learning (FL) is the network communication cost of sending model updates from client devices to the central server. We present a comprehensive empirical study of the statistics of model updates in FL, as well as the role and benefits of various compression techniques. Motivated by these observations, we propose a novel method to reduce the average communication cost, which is near-optimal in many use cases, and outperforms Top-K, DRIVE, 3LC and QSGD on Stack Overflow next-word prediction, a realistic and challenging FL benchmark. This is achieved by examining the problem using rate-distortion theory, and proposing distortion as a reliable proxy for model accuracy. Distortion can be more effectively used for optimizing the trade-off between model performance and communication cost across clients. We demonstrate empirically that in spite of the non-i.i.d. nature of federated learning, the rate-distortion frontier is consistent across datasets, optimizers, clients and training rounds.

Key facts. Cited by 14 papers in the map (data through 14 Oct 2026). Builds on 29 of its 51 references that are in the map. First posted on arXiv 7 Jan 2022.

Builds on

29 of 51 references.

References · 29

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  3. 2019Advances and Open Problems in Federated Learningcited by 3,046
  4. 2018Federated Optimization in Heterogeneous Networkscited by 2,987
  5. 2018Group Normalizationcited by 1,524
  6. 2016Federated Learning: Strategies for Improving Communication Efficiencycited by 1,475
  7. 2020Adaptive Federated Optimizationcited by 1,046
  8. 2019Towards Federated Learning at Scale: System Designcited by 832
  9. 2017Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Trainingcited by 478
  10. 2019Robust and Communication-Efficient Federated Learning from Non-IID Datacited by 419
  11. 2019FedPAQ: A Communication-Efficient Federated Learning Method with Periodic Averaging and Quantizationcited by 356
  12. 2021A Field Guide to Federated Optimizationcited by 315
17 more
  1. 2019The Non-IID Data Quagmire of Decentralized Machine Learningcited by 305
  2. 2017TernGrad: Ternary Gradients to Reduce Communication in Distributed Deep Learningcited by 256
  3. 2017Sparse Communication for Distributed Gradient Descentcited by 249
  4. 2019PowerSGD: Practical Low-Rank Gradient Compression for Distributed Optimizationcited by 238
  5. 2020FetchSGD: Communication-Efficient Federated Learning with Sketchingcited by 179
  6. 2020Federated Learning with Compression: Unified Analysis and Sharp Guaranteescited by 160
  7. 2016QSGD: Communication-Efficient SGD via Gradient Quantization and Encodingcited by 147
  8. 2020On Biased Compression for Distributed Learningcited by 140
  9. 2016Distributed Mean Estimation with Limited Communicationcited by 136
  10. 2018ATOMO: Communication-efficient Learning via Atomic Sparsificationcited by 136
  11. 2021Papaya: Practical, Private, and Scalable Federated Learningcited by 90
  12. 2021On Large-Cohort Training for Federated Learningcited by 74
  13. 2020IBM Federated Learning: an Enterprise Framework White Paper V0.1cited by 57
  14. 2021DRIVE: One-bit Distributed Mean Estimationcited by 52
  15. 2020Optimal Gradient Compression for Distributed and Federated Learningcited by 39
  16. 2020FedSKETCH: Communication-Efficient and Private Federated Learning via Sketchingcited by 20
  17. 20183LC: Lightweight and Effective Traffic Compression for Distributed Machine Learningcited by 16

Built on by

Cited by year

Citing papers per year: 2022: 7; 2023: 1; 2024: 4; 2025: 2.

14 papers.

Sort:

  1. 2022Sparse Random Networks for Communication-Efficient Federated Learningcited by 32
  2. 2022Correlated quantization for distributed mean estimation and optimizationcited by 14
  3. 2025Overcoming the Communication-Performance Tradeoff in LLM Pretrainingcited by 12
  4. 2022Federated Select: A Primitive for Communication- and Memory-Efficient Federated Learningcited by 11
  5. 2022Linear Stochastic Bandits over a Bit-Constrained Channelcited by 9
  6. 2022Mixed Federated Learning: Joint Decentralized and Centralized Learningcited by 8
  7. 2022QUIC-FL: Quick Unbiased Compression for Federated Learningcited by 7
  8. 2022ResFed: Communication Efficient Federated Learning by Transmitting Deep Compressed Residualscited by 4
  9. 2024Improved Communication-Privacy Trade-offs in L2 Mean Estimation under Streaming Differential Privacycited by 4
  10. 2024SplitFedZip: Learned Compression for Data Transfer Reduction in Split-Federated Learningcited by 4
  11. 2025PRISM: Privacy-Preserving Improved Stochastic Masking for Federated Generative Modelscited by 3
  12. 2023Federated Automatic Differentiationcited by 2
2 more
  1. 2024Theoretical Analysis of Privacy Leakage in Trustworthy Federated Learning: A Perspective from Linear Algebra and Optimization Theorycited by 2
  2. 2024Towards Communication-efficient Federated Learning via Sparse and Aligned Adaptive Optimizationcited by 1