mlsanityData to 7 Oct 2026

LASER: Linear Compression in Wireless Distributed Optimization

arXiv 2310.13033 ↗First posted 19 Oct 2023cs.NE cs.AI cs.ITcited by 6

Not on the map. Why?

Abstract (arXiv)

Data-parallel SGD is the de facto algorithm for distributed optimization, especially for large scale machine learning. Despite its merits, communication bottleneck is one of its persistent issues. Most compression schemes to alleviate this either assume noiseless communication links, or fail to achieve good performance on practical tasks. In this paper, we close this gap and introduce LASER: LineAr CompreSsion in WirEless DistRibuted Optimization. LASER capitalizes on the inherent low-rank structure of gradients and transmits them efficiently over the noisy channels. Whilst enjoying theoretical guarantees similar to those of the classical SGD, LASER shows consistent gains over baselines on a variety of practical benchmarks. In particular, it outperforms the state-of-the-art compression schemes on challenging computer vision and GPT language modeling tasks. On the latter, we obtain 50-64 % improvement in perplexity over our baselines for noisy channels.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 30 of its 57 references that are in the map. First posted on arXiv 19 Oct 2023.

Builds on

30 of 57 references.

References · 30

  1. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  2. 2016Federated Learning: Strategies for Improving Communication Efficiencycited by 1,461
  3. 2016Optimization Methods for Large-Scale Machine Learningcited by 1,140
  4. 2016Federated Optimization: Distributed Machine Learning for On-Device Intelligencecited by 558
  5. 2018Sparsified SGD with Memorycited by 373
  6. 2019Error Feedback Fixes SignSGD and other Gradient Compression Schemescited by 306
  7. 2017TernGrad: Ternary Gradients to Reduce Communication in Distributed Deep Learningcited by 253
  8. 2017Sparse Communication for Distributed Gradient Descentcited by 243
  9. 2019PowerSGD: Practical Low-Rank Gradient Compression for Distributed Optimizationcited by 234
  10. 2018Implicit Self-Regularization in Deep Neural Networks: Evidence from Random Matrix Theory and Implications for Learningcited by 211
  11. 2017Gradient Sparsification for Communication-Efficient Distributed Optimizationcited by 199
  12. 2018Broadband Analog Aggregation for Low-Latency Federated Edge Learning (Extended Version)cited by 186
18 more
  1. 2019Qsparse-local-SGD: Distributed SGD with Quantization, Sparsification, and Local Computationscited by 179
  2. 2017Spectral Norm Regularization for Improving the Generalizability of Deep Learningcited by 178
  3. 2019Federated Learning over Wireless Fading Channelscited by 178
  4. 2020FetchSGD: Communication-Efficient Federated Learning with Sketchingcited by 178
  5. 2018ATOMO: Communication-efficient Learning via Atomic Sparsificationcited by 135
  6. 2020Communication-Efficient Edge AI: Algorithms and Systemscited by 95
  7. 2019Natural Compression for Distributed Deep Learningcited by 88
  8. 2019Communication-efficient distributed SGD with Sketchingcited by 81
  9. 20158-Bit Approximations for Parallelism in Deep Learningcited by 56
  10. 2021EDEN: Communication-Efficient and Robust Distributed Mean Estimation for Federated Learningcited by 53
  11. 2021DRIVE: One-bit Distributed Mean Estimationcited by 52
  12. 2022Sparse Random Networks for Communication-Efficient Federated Learningcited by 32
  13. 2018Variance-based Gradient Compression for Efficient Distributed Deep Learningcited by 21
  14. 2018GradiVeQ: Vector Quantization for Bandwidth-Efficient Gradient Aggregation in Distributed CNN Trainingcited by 21
  15. 2020FedSKETCH: Communication-Efficient and Private Federated Learning via Sketchingcited by 20
  16. 20183LC: Lightweight and Effective Traffic Compression for Distributed Machine Learningcited by 16
  17. 2020Decentralized optimization over noisy, rate-constrained networks: Achieving consensus by communicating differencescited by 12
  18. 2019Exploring Fast and Communication-Efficient Algorithms in Large-scale Distributed Networkscited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 3; 2026: 1.

6 papers.

  1. 2025Mitigating Unintended Memorization with LoRA in Federated Learning for LLMscited by 6
  2. 2024Communication-Efficient Byzantine-Resilient Federated Zero-Order Optimizationcited by 4
  3. 2025Noise Resilient Over-The-Air Federated Learning In Heterogeneous Wireless Networkscited by 1
  4. 2024Efficient Wireless Federated Learning via Low-Rank Gradient Factorizationcited by 0
  5. 2025Timely Parameter Updating in Over-the-Air Federated Learningcited by 0
  6. 2026Exploiting Correlations in Federated Learning: Opportunities and Practical Limitationscited by 0