mlsanityData to 14 Oct 2026

TernGrad: Ternary Gradients to Reduce Communication in Distributed Deep Learning

arXiv 1705.07878 ↗First posted 22 May 2017cs.LG cs.DC cs.NEcited by 256

Not on the map. Why?

Abstract (arXiv)

High network communication cost for synchronizing gradients and parameters is the well-known bottleneck of distributed training. In this work, we propose TernGrad that uses ternary gradients to accelerate distributed deep learning in data parallelism. Our approach requires only three numerical levels {-1,0,1}, which can aggressively reduce the communication time. We mathematically prove the convergence of TernGrad under the assumption of a bound on gradients. Guided by the bound, we propose layer-wise ternarizing and gradient clipping to improve its convergence. Our experiments show that applying TernGrad on AlexNet does not incur any accuracy loss and can even improve accuracy. The accuracy loss of GoogLeNet induced by TernGrad is less than 2% on average. Finally, a performance model is proposed to study the scalability of TernGrad. Experiments show significant speed gains for various deep neural networks. Our source code is available.

Key facts. Cited by 256 papers in the map (data through 14 Oct 2026). First posted on arXiv 22 May 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2020: 1; 2021: 82; 2022: 54; 2023: 51; 2024: 36; 2025: 17; 2026: 15.

256 papers.

Sort:

By area:

  1. 2021Federated Learning on Non-IID Data: A Surveycited by 346
  2. 2021A Field Guide to Federated Optimizationcited by 315
  3. 2021FedPara: Low-Rank Hadamard Product for Communication-Efficient Federated Learningcited by 141
  4. 2021Distributed Learning in Wireless Networks: Recent Progress and Future Challengescited by 129
  5. 2021Linear Convergence in Federated Learning: Tackling Client Heterogeneity and Sparse Gradientscited by 123
  6. 2023Stochastic Controlled Averaging for Federated Learning with Communication Compressioncited by 101
  7. 2021Edge Artificial Intelligence for 6G: Vision, Enabling Technologies, and Applicationscited by 97
  8. 2024What is Your Data Worth to GPT? LLM-Scale Data Valuation with Influence Functionscited by 94
  9. 20211-bit Adam: Communication Efficient Large-Scale Training with Adam's Convergence Speedcited by 75
  10. 2022DisPFL: Towards Communication-Efficient Personalized Federated Learning via Decentralized Sparse Trainingcited by 74
  11. 2021Provably Secure Federated Learning against Malicious Clientscited by 63
  12. 2021Adaptive Quantization of Model Updates for Communication-Efficient Federated Learningcited by 63
38 more
  1. 2021Pufferfish: Communication-efficient Models At No Extra Costcited by 61
  2. 2024FLUX: Fast Software-based Communication Overlap On GPUs Through Kernel Fusioncited by 57
  3. 2021EDEN: Communication-Efficient and Robust Distributed Mean Estimation for Federated Learningcited by 55
  4. 2021DRIVE: One-bit Distributed Mean Estimationcited by 52
  5. 2022PipeGCN: Efficient Full-Graph Training of Graph Convolutional Networks with Pipelined Feature Communicationcited by 45
  6. 2021Rethinking gradient sparsification as total error minimizationcited by 41
  7. 2024Memory-Efficient LLM Training with Online Subspace Descentcited by 39
  8. 2022Stochastic Gradient Descent-Ascent: Unified Theory and New Efficient Methodscited by 37
  9. 2021Moshpit SGD: Communication-Efficient Decentralized Training on Heterogeneous Unreliable Devicescited by 36
  10. 2021Layer-wise Adaptive Model Aggregation for Scalable Federated Learningcited by 36
  11. 2022Compressed-VFL: Communication-Efficient Learning with Vertically Partitioned Datacited by 36
  12. 2022Lower Bounds and Nearly Optimal Algorithms in Distributed Learning with Communication Compressioncited by 33
  13. 2021An Efficient Statistical-based Gradient Compression Technique for Distributed Training Systemscited by 32
  14. 2022Enable Deep Learning on Mobile Devices: Methods, Systems, and Applicationscited by 32
  15. 2022Sparse Random Networks for Communication-Efficient Federated Learningcited by 32
  16. 2022FedSynth: Gradient Compression via Synthetic Data in Federated Learningcited by 31
  17. 2022Fine-tuning Language Models over Slow Networks using Activation Compression with Guaranteescited by 30
  18. 2023DoCoFL: Downlink Compression for Cross-Device Federated Learningcited by 28
  19. 2021FEDZIP: A Compression Framework for Communication-Efficient Federated Learningcited by 27
  20. 2021ProgFed: Effective, Communication, and Computation Efficient Federated Learning by Progressive Trainingcited by 27
  21. 2021Time-Correlated Sparsification for Communication-Efficient Federated Learningcited by 26
  22. 2021On the Utility of Gradient Compression in Distributed Training Systemscited by 26
  23. 2021Distributed Second Order Methods with Fast Rates and Compressed Communicationcited by 25
  24. 2021Toward Communication Efficient Adaptive Gradient Methodcited by 25
  25. 2022Scaling Language Model Size in Cross-Device Federated Learningcited by 25
  26. 20211-bit LAMB: Communication Efficient Large-Scale Large-Batch Training with LAMB's Convergence Speedcited by 24
  27. 2022Near-Optimal Sparse Allreduce for Distributed Deep Learningcited by 23
  28. 2021Optimal Compression of Locally Differentially Private Mechanismscited by 22
  29. 2022Egeria: Efficient DNN Training with Knowledge-Guided Layer Freezingcited by 22
  30. 2023Privacy Amplification via Compression: Achieving the Optimal Privacy-Accuracy-Communication Trade-off in Distributed Mean Estimationcited by 22
  31. 2023Reducing Communication for Split Learning by Randomized Top-k Sparsificationcited by 22
  32. 2021Smoothness Matrices Beat Smoothness Constants: Better Communication Compression Techniques for Distributed Optimizationcited by 21
  33. 2021Preserved central model for faster bidirectional compression in distributed settingscited by 21
  34. 2022FedShuffle: Recipes for Better Use of Local Work in Federated Learningcited by 21
  35. 2021Learned Gradient Compression for Distributed Deep Learningcited by 20
  36. 2021Trade-offs of Local SGD at Scale: An Empirical Studycited by 20
  37. 2023FusionAI: Decentralized Training and Deploying LLMs with Massive Consumer-Level GPUscited by 20
  38. 2021CPT: Efficient Deep Neural Network Training via Cyclic Precisioncited by 19

50 most cited of 256 All 256 (JSON)