mlsanityData to 13 Oct 2026

Lion Cub: Minimizing Communication Overhead in Distributed Lion

arXiv 2411.16462 ↗First posted 25 Nov 2024cs.LG cs.DCcited by 1

Not on the map. Why?

Abstract (arXiv)

Communication overhead is a key challenge in distributed deep learning, especially on slower Ethernet interconnects, and given current hardware trends, communication is likely to become a major bottleneck. While gradient compression techniques have been explored for SGD and Adam, the Lion optimizer has the distinct advantage that its update vectors are the output of a sign operation, enabling straightforward quantization. However, simply compressing updates for communication and using techniques like majority voting fails to lead to end-to-end speedups due to inefficient communication algorithms and reduced convergence. We analyze three factors critical to distributed learning with Lion: optimizing communication methods, identifying effective quantization methods, and assessing the necessity of momentum synchronization. Our findings show that quantization techniques adapted to Lion and selective momentum synchronization can significantly reduce communication costs while maintaining convergence. We combine these into Lion Cub, which enables up to 5x speedups in end-to-end training compared to Lion. This highlights Lion's potential as a communication-efficient solution for distributed training.

Key facts. Cited by 1 paper in the map (data through 13 Oct 2026). Builds on 35 of its 55 references that are in the map. First posted on arXiv 25 Nov 2024.

Builds on

35 of 55 references.

References · 35

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,950
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,223
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,979
  4. 2020Scaling Laws for Neural Language Modelscited by 6,494
  5. 2020Training data-efficient image transformers & distillation through attentioncited by 4,574
  6. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,388
  7. 2022Training Compute-Optimal Large Language Modelscited by 2,972
  8. 2017Improved Regularization of Convolutional Neural Networks with Cutoutcited by 1,627
  9. 2018signSGD: Compressed Optimisation for Non-Convex Problemscited by 567
  10. 2017Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Trainingcited by 478
  11. 2023Symbolic Discovery of Optimization Algorithmscited by 425
  12. 20218-bit Optimizers via Block-wise Quantizationcited by 399
23 more
  1. 2017TernGrad: Ternary Gradients to Reduce Communication in Distributed Deep Learningcited by 256
  2. 2018ATOMO: Communication-efficient Learning via Atomic Sparsificationcited by 136
  3. 2023Noise Is Not the Main Factor Behind the Gap Between SGD and Adam on Transformers, but Sign Descent Might Becited by 114
  4. 2020Stochastic Optimization with Heavy-Tailed Noise via Accelerated Gradient Clippingcited by 104
  5. 2019Communication-Efficient Federated Deep Learning with Asynchronous Model Update and Temporally Weighted Aggregationcited by 100
  6. 2022Petals: Collaborative Inference and Fine-tuning of Large Modelscited by 87
  7. 20211-bit Adam: Communication Efficient Large-Scale Training with Adam's Convergence Speedcited by 74
  8. 2021Distributed Deep Learning in Open Collaborationscited by 58
  9. 2019Stochastic Sign Descent Methods: New Algorithms and Better Theorycited by 49
  10. 2021Scaling Federated Learning for Fine-tuning of Large Language Modelscited by 36
  11. 2021Moshpit SGD: Communication-Efficient Decentralized Training on Heterogeneous Unreliable Devicescited by 36
  12. 2021Layer-wise Adaptive Model Aggregation for Scalable Federated Learningcited by 36
  13. 2018SparCML: High-Performance Sparse Communication for Machine Learningcited by 34
  14. 2020Synthesizing Optimal Collective Algorithmscited by 30
  15. 2020Communication-Efficient Distributed Deep Learning: A Comprehensive Surveycited by 26
  16. 20211-bit LAMB: Communication Efficient Large-Scale Large-Batch Training with LAMB's Convergence Speedcited by 24
  17. 2020Is Network the Bottleneck of Distributed Training?cited by 20
  18. 2024Communication Efficient Distributed Training with Distributed Lioncited by 18
  19. 2022Maximizing Communication Efficiency for Large-scale Training via 0/1 Adamcited by 15
  20. 2022Taming Fat-Tailed ("Heavier-Tailed'' with Potentially Infinite Variance) Noise in Federated Learningcited by 10
  21. 2023EPISODE: Episodic Gradient Clipping with Periodic Resampled Corrections for Federated Learning with Heterogeneous Datacited by 7
  22. 2024Why Go Full? Elevating Federated Learning Through Partial Network Updatescited by 6
  23. 2024Improved Quantization Strategies for Managing Heavy-tailed Gradients in Distributed Learningcited by 1

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025Convergence Analysis of the Lion Optimizer in Centralized and Distributed Settingscited by 6

Similar, not cited

Close in wording, no citation either way

  1. 2021Unbiased Single-scale and Multi-scale Quantizers for Distributed Optimization
  2. 2021Peering Beyond the Gradient Veil with Distributed Auto Differentiation
  3. 2023Optimizing Distributed ML Communication with Fused Computation-Collective Operations
  4. 2023Quantize Once, Train Fast: Allreduce-Compatible Compression with Provable Guarantees
  5. 2024SDP4Bit: Toward 4-bit Communication Quantization in Sharded Data Parallelism for LLM Training
  6. 2023Evaluation and Optimization of Gradient Compression for Distributed Deep Learning
  7. 2026LoDAdaC: a unified local training-based decentralized framework with adaptive gradients and compressed communication
  8. 2021Learned Gradient Compression for Distributed Deep Learning