Gradient Descent with Compressed Iterates
Not on the map. Why?
Abstract (arXiv)
We propose and analyze a new type of stochastic first order method: gradient descent with compressed iterates (GDCI). GDCI in each iteration first compresses the current iterate using a lossy randomized compression technique, and subsequently takes a gradient step. This method is a distillation of a key ingredient in the current practice of federated learning, where a model needs to be compressed by a mobile device before it is sent back to a server for aggregation. Our analysis provides a step towards closing the gap between the theory and practice of federated learning, and opens the possibility for many extensions.
Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). First posted on arXiv 10 Sep 2019.
Builds on
No reference list yet.
Built on by
Citing papers per year: 2021: 3; 2022: 3; 2023: 4; 2024: 0; 2025: 1.
11 papers.
- 2022EF-BV: A Unified Theory of Error Feedback and Variance Reduction Mechanisms for Biased and Unbiased Compression in Distributed Optimizationcited by 17
- 2021Communication-Efficient Federated Learning with Binary Neural Networkscited by 13
- 2022Federated Random Reshuffling with Compression and Variance Reductioncited by 11
- 2023Knowledge Distillation Performs Partial Variance Reductioncited by 11
- 2023Improving Accelerated Federated Learning with Compression and Importance Samplingcited by 7
- 2021EF21 with Bells & Whistles: Six Algorithmic Extensions of Modern Error Feedbackcited by 6
- 2022Shifted Compression Framework: Generalizations and Improvementscited by 6
- 2023Towards a Better Theoretical Understanding of Independent Subnetwork Trainingcited by 6
- 2021RingFed: Reducing Communication Costs in Federated Learning on Non-IID Datacited by 4
- 2023Federated Learning over Hierarchical Wireless Networks: Training Latency Minimization via Submodel Partitioningcited by 1
- 2025First Provable Guarantees for Practical Private FL: Beyond Restrictive Assumptionscited by 0
- 2025First Provable Guarantees for Practical Private FL: Beyond Restrictive Assumptions25 Dec 2025
- 2023Federated Learning over Hierarchical Wireless Networks: Training Latency Minimization via Submodel Partitioning27 Oct 2023
- 2023Towards a Better Theoretical Understanding of Independent Subnetwork Training28 Jun 2023
- 2023Improving Accelerated Federated Learning with Compression and Importance Sampling5 Jun 2023
- 2023Knowledge Distillation Performs Partial Variance Reduction27 May 2023
- 2022Shifted Compression Framework: Generalizations and Improvements21 Jun 2022
- 2022EF-BV: A Unified Theory of Error Feedback and Variance Reduction Mechanisms for Biased and Unbiased Compression in Distributed Optimization9 May 2022
- 2022Federated Random Reshuffling with Compression and Variance Reduction8 May 2022
- 2021EF21 with Bells & Whistles: Six Algorithmic Extensions of Modern Error Feedback7 Oct 2021
- 2021Communication-Efficient Federated Learning with Binary Neural Networks5 Oct 2021
- 2021RingFed: Reducing Communication Costs in Federated Learning on Non-IID Data19 Jul 2021