mlsanityData to 7 Oct 2026

ATOMO: Communication-efficient Learning via Atomic Sparsification

arXiv 1806.04090 ↗First posted 11 Jun 2018stat.ML cs.DC cs.LGcited by 135

Not on the map. Why?

Abstract (arXiv)

Distributed model training suffers from communication overheads due to frequent gradient updates transmitted between compute nodes. To mitigate these overheads, several studies propose the use of sparsified stochastic gradients. We argue that these are facets of a general sparsification method that can operate on any possible atomic decomposition. Notable examples include element-wise, singular value, and Fourier decompositions. We present ATOMO, a general framework for atomic sparsification of stochastic gradients. Given a gradient, an atomic decomposition, and a sparsity budget, ATOMO gives a random unbiased sparsification of the atoms minimizing variance. We show that recent methods such as QSGD and TernGrad are special cases of ATOMO and that sparsifiying the singular value decomposition of neural networks gradients, rather than their coordinates, can lead to significantly faster distributed training.

Key facts. Cited by 135 papers in the map (data through 7 Oct 2026). First posted on arXiv 11 Jun 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 39; 2022: 28; 2023: 20; 2024: 23; 2025: 13; 2026: 12.

135 papers.

Sort:

  1. 2024GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projectioncited by 447
  2. 2021A Field Guide to Federated Optimizationcited by 312
  3. 2021FjORD: Fair and Accurate Federated Learning under heterogeneous targets with Ordered Dropoutcited by 192
  4. 2021Edge Artificial Intelligence for 6G: Vision, Enabling Technologies, and Applicationscited by 96
  5. 2021Large Scale Private Learning via Low-rank Reparametrizationcited by 87
  6. 2025Dion: Distributed Orthonormalized Updatescited by 64
  7. 2021Pufferfish: Communication-efficient Models At No Extra Costcited by 61
  8. 2021Adaptive Quantization of Model Updates for Communication-Efficient Federated Learningcited by 60
  9. 2025ASGO: Adaptive Structured Gradient Optimizationcited by 55
  10. 2021DRIVE: One-bit Distributed Mean Estimationcited by 52
  11. 2021Hierarchical Federated Learning with Quantization: Convergence Analysis and System Designcited by 47
  12. 2022ZeroFL: Efficient On-Device Training for Federated Learning with Local Sparsitycited by 45
38 more
  1. 2021Rethinking gradient sparsification as total error minimizationcited by 41
  2. 2023FedML-HE: An Efficient Homomorphic-Encryption-Based Privacy-Preserving Federated Learning Systemcited by 38
  3. 2022No One Left Behind: Inclusive Federated Learning over Heterogeneous Devicescited by 36
  4. 2021Layer-wise Adaptive Model Aggregation for Scalable Federated Learningcited by 35
  5. 2022SoteriaFL: A Unified Framework for Private Federated Learning with Communication Compressioncited by 34
  6. 2022Federated Learning with Sparsified Model Perturbation: Improving Accuracy under Client-Level Differential Privacycited by 32
  7. 2022Sparse Random Networks for Communication-Efficient Federated Learningcited by 32
  8. 2022Fine-tuning Language Models over Slow Networks using Activation Compression with Guaranteescited by 29
  9. 2021FEDZIP: A Compression Framework for Communication-Efficient Federated Learningcited by 27
  10. 2023A Survey on Decentralized Federated Learningcited by 27
  11. 2021On the Utility of Gradient Compression in Distributed Training Systemscited by 26
  12. 2022Towards Federated Learning against Noisy Labels via Local Self-Regularizationcited by 26
  13. 2022A Survey on Federated Recommendation Systemscited by 25
  14. 2021An Information-Theoretic Justification for Model Pruningcited by 23
  15. 2021Optimal Compression of Locally Differentially Private Mechanismscited by 22
  16. 2023Reducing Communication for Split Learning by Randomized Top-k Sparsificationcited by 22
  17. 2021Smoothness Matrices Beat Smoothness Constants: Better Communication Compression Techniques for Distributed Optimizationcited by 21
  18. 2022FedShuffle: Recipes for Better Use of Local Work in Federated Learningcited by 19
  19. 2022Communication Acceleration of Local Gradient Methods via an Accelerated Primal-Dual Algorithm with Inexact Proxcited by 19
  20. 2021Fast Federated Learning by Balancing Communication Trade-Offscited by 18
  21. 2022Scalable and Low-Latency Federated Learning with Cooperative Mobile Edge Networkingcited by 18
  22. 2023ComPEFT: Compression for Communicating Parameter Efficient Updates via Sparsification and Quantizationcited by 18
  23. 2021Distributed Learning Systems with First-order Methodscited by 17
  24. 2021On the Convergence of Decentralized Adaptive Gradient Methodscited by 17
  25. 2021Persia: An Open, Hybrid System Scaling Deep Learning-based Recommenders up to 100 Trillion Parameterscited by 17
  26. 2022A General Theory for Federated Optimization with Asynchronous and Heterogeneous Clients Updatescited by 17
  27. 2022GradSkip: Communication-Accelerated Local Gradient Methods with Better Computational Complexitycited by 17
  28. 2024Grass: Compute Efficient Low-Memory LLM Training with Structured Sparse Gradientscited by 17
  29. 2021Towards Demystifying Serverless Machine Learning Trainingcited by 16
  30. 2022Recycling Model Updates in Federated Learning: Are Gradient Subspaces Low-Rank?cited by 16
  31. 2022Maximizing Communication Efficiency for Large-scale Training via 0/1 Adamcited by 15
  32. 2022Optimizing the Communication-Accuracy Trade-off in Federated Learning with Rate-Distortion Theorycited by 14
  33. 2022FedLite: A Scalable Approach for Federated Learning on Resource-constrained Clientscited by 14
  34. 2022Correlated quantization for distributed mean estimation and optimizationcited by 14
  35. 2023A Survey on Approximate Edge AI for Energy Efficient Autonomous Driving Servicescited by 14
  36. 2021Neural Distributed Source Codingcited by 13
  37. 2021BAGUA: Scaling up Distributed Learning with System Relaxationscited by 13
  38. 2023Quantized Distributed Training of Large Models with Convergence Guaranteescited by 13

50 most cited of 135 All 135 (JSON)