mlsanityData to 14 Oct 2026

Sign Bit is Enough: A Learning Synchronization Framework for Multi-hop All-reduce with Ultimate Compression

arXiv 2204.06787 ↗First posted 14 Apr 2022cs.LG cs.DCcited by 5

Not on the map. Why?

Abstract (arXiv)

Traditional one-bit compressed stochastic gradient descent can not be directly employed in multi-hop all-reduce, a widely adopted distributed training paradigm in network-intensive high-performance computing systems such as public clouds. According to our theoretical findings, due to the cascading compression, the training process has considerable deterioration on the convergence performance. To overcome this limitation, we implement a sign-bit compression-based learning synchronization framework, Marsit. It prevents cascading compression via an elaborate bit-wise operation for unbiased sign aggregation and its specific global compensation mechanism for mitigating compression deviation. The proposed framework retains the same theoretical convergence rate as non-compression mechanisms. Experimental results demonstrate that Marsit reduces up to 35% training time while preserving the same accuracy as training without compression.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 17 of its 31 references that are in the map. First posted on arXiv 14 Apr 2022.

Builds on

17 of 31 references.

References · 17

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  4. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  5. 2018signSGD: Compressed Optimisation for Non-Convex Problemscited by 571
  6. 2019Error Feedback Fixes SignSGD and other Gradient Compression Schemescited by 310
  7. 2018Horovod: fast and easy distributed deep learning in TensorFlowcited by 279
  8. 2017TernGrad: Ternary Gradients to Reduce Communication in Distributed Deep Learningcited by 256
  9. 2019PowerSGD: Practical Low-Rank Gradient Compression for Distributed Optimizationcited by 238
  10. 2017Gradient Sparsification for Communication-Efficient Distributed Optimizationcited by 201
  11. 20211-bit Adam: Communication Efficient Large-Scale Training with Adam's Convergence Speedcited by 75
  12. 2021Quasi-Global Momentum: Accelerating Decentralized Deep Learning on Heterogeneous Datacited by 69
5 more
  1. 2018Highly Scalable Deep Learning Training System with Mixed-Precision: Training ImageNet in Four Minutescited by 63
  2. 2020Optimal Complexity in Decentralized Trainingcited by 60
  3. 2019Stochastic Sign Descent Methods: New Algorithms and Better Theorycited by 49
  4. 2021Accelerating Gossip SGD with Periodic Global Averagingcited by 25
  5. 2018GradiVeQ: Vector Quantization for Bandwidth-Efficient Gradient Aggregation in Distributed CNN Trainingcited by 21

Built on by

Cited by year

Citing papers per year: 2022: 1; 2023: 0; 2024: 1; 2025: 3.

5 papers.

  1. 2024FIARSE: Model-Heterogeneous Federated Learning via Importance-Aware Submodel Extractioncited by 21
  2. 2025Low-Precision Training of Large Language Models: Methods, Challenges, and Opportunitiescited by 11
  3. 2022GlueFL: Reconciling Client Sampling and Model Masking for Bandwidth Efficient Federated Learningcited by 7
  4. 2025FedFetch: Faster Federated Learning with Adaptive Downstream Prefetchingcited by 2
  5. 2025Towards Privacy-Preserving and Heterogeneity-aware Split Federated Learning via Probabilistic Maskingcited by 0