mlsanityData to 7 Oct 2026

Fine-tuning Language Models over Slow Networks using Activation Compression with Guarantees

arXiv 2206.01299 ↗First posted 2 Jun 2022cs.LG cs.DCcited by 29

Not on the map. Why?

Abstract (arXiv)

Communication compression is a crucial technique for modern distributed learning systems to alleviate their communication bottlenecks over slower networks. Despite recent intensive studies of gradient compression for data parallel-style training, compressing the activations for models trained with pipeline parallelism is still an open problem. In this paper, we propose AC-SGD, a novel activation compression algorithm for communication-efficient pipeline parallelism training over slow networks. Different from previous efforts in activation compression, instead of compressing activation values directly, AC-SGD compresses the changes of the activations. This allows us to show, to the best of our knowledge for the first time, that one can still achieve O(1/√T) convergence rate for non-convex objectives under activation compression, without making assumptions on gradient unbiasedness that do not hold for deep learning models with non-linear activation functions.We then show that AC-SGD can be optimized and implemented efficiently, without additional end-to-end runtime overhead.We evaluated AC-SGD to fine-tune language models with up to 1.5 billion parameters, compressing activations to 2-4 bits.AC-SGD provides up to 4.3X end-to-end speed-up in slower networks, without sacrificing model quality. Moreover, we also show that AC-SGD can be combined with state-of-the-art gradient compression algorithms to enable "end-to-end communication compression: All communications between machines, including model gradients, forward activations, and backward gradients are compressed into lower precision.This provides up to 4.9X end-to-end speed-up, without sacrificing model quality.

Key facts. Cited by 29 papers in the map (data through 7 Oct 2026). Builds on 59 of its 89 references that are in the map. First posted on arXiv 2 Jun 2022.

Builds on

59 of 89 references.

References · 59

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  6. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,291
  7. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  8. 2016Optimization Methods for Large-Scale Machine Learningcited by 1,140
  9. 2018signSGD: Compressed Optimisation for Non-Convex Problemscited by 561
  10. 2021Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networkscited by 521
  11. 2017Can Decentralized Algorithms Outperform Centralized Algorithms? A Case Study for Decentralized Parallel Stochastic Gradient Descentcited by 496
  12. 2016Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activationscited by 413
38 more
  1. 2017Scalable Training of Artificial Neural Networks with Adaptive Sparse Connectivity inspired by Network Sciencecited by 325
  2. 2018Horovod: fast and easy distributed deep learning in TensorFlowcited by 275
  3. 2017TernGrad: Ternary Gradients to Reduce Communication in Distributed Deep Learningcited by 253
  4. 2018The Convergence of Sparsified Gradient Methodscited by 228
  5. 2019Decentralized Stochastic Optimization and Gossip Algorithms with Compressed Communicationcited by 218
  6. 2019Sparse Networks from Scratch: Faster Training without Losing Performancecited by 210
  7. 2017Gradient Sparsification for Communication-Efficient Distributed Optimizationcited by 199
  8. 2019Parameter Efficient Training of Deep Convolutional Neural Networks by Dynamic Sparse Reparameterizationcited by 187
  9. 2020FetchSGD: Communication-Efficient Federated Learning with Sketchingcited by 178
  10. 2017Asynchronous Decentralized Parallel Stochastic Gradient Descentcited by 159
  11. 2018D2: Decentralized Training over Decentralized Datacited by 140
  12. 2018ATOMO: Communication-efficient Learning via Atomic Sparsificationcited by 135
  13. 2020Memory-Efficient Pipeline-Parallel DNN Trainingcited by 133
  14. 2019DoubleSqueeze: Parallel Stochastic Gradient Descent with Double-Pass Error-Compensated Compressioncited by 98
  15. 2020Acceleration for Compressed Gradient Descent in Distributed and Federated Optimizationcited by 90
  16. 2018Communication Compression for Decentralized Trainingcited by 87
  17. 2021Sparse Training via Boosting Pruning Plasticity with Neuroregenerationcited by 83
  18. 2019Communication-efficient distributed SGD with Sketchingcited by 81
  19. 2020On the Convergence of SGD with Biased Gradientscited by 79
  20. 2021MARINA: Faster Non-Convex Distributed Learning with Compressioncited by 77
  21. 2021Pixelated Butterfly: Simple and Efficient Sparse training for Neural Network Modelscited by 75
  22. 20211-bit Adam: Communication Efficient Large-Scale Training with Adam's Convergence Speedcited by 74
  23. 2020Dynamic Sparse Training: Find Efficient Sparse Network From Scratch With Trainable Masked Layerscited by 66
  24. 2021Pufferfish: Communication-efficient Models At No Extra Costcited by 61
  25. 2021ActNN: Reducing Training Memory Footprint via 2-Bit Activation Compressed Trainingcited by 61
  26. 2021Distributed Deep Learning in Open Collaborationscited by 58
  27. 2019PipeMare: Asynchronous Pipeline Parallel DNN Trainingcited by 54
  28. 2019Stochastic Sign Descent Methods: New Algorithms and Better Theorycited by 49
  29. 2020Sparse Weight Activation Trainingcited by 45
  30. 2020Error Compensated Distributed SGD Can Be Acceleratedcited by 45
  31. 2020Towards Crowdsourced Training of Large Neural Networks using Decentralized Mixture-of-Expertscited by 43
  32. 2020Adaptive Gradient Quantization for Data-Parallel SGDcited by 43
  33. 2016Efficient Distributed Learning with Sparsitycited by 38
  34. 2017Compressing DMA Engine: Leveraging Activation Sparsity for Training Deep Neural Networkscited by 36
  35. 2018Accelerating Convolutional Neural Networks via Activation Map Compressioncited by 34
  36. 2021An Efficient Statistical-based Gradient Compression Technique for Distributed Training Systemscited by 32
  37. 2018Pipe-SGD: A Decentralized Pipelined SGD Framework for Distributed Deep Net Trainingcited by 28
  38. 2020CSER: Communication-efficient SGD with Error Resetcited by 27

50 most cited of 59 All 59 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 2; 2023: 9; 2024: 4; 2025: 9; 2026: 5.

29 papers.

Sort:

By area:

  1. 2023Fast Distributed Inference Serving for Large Language Modelscited by 143
  2. 2023Scaling Expert Language Models with Unsupervised Domain Discoverycited by 59
  3. 2023SWARM Parallelism: Training Large Models Can Be Surprisingly Communication-Efficientcited by 58
  4. 2023Distributed Inference and Fine-tuning of Large Language Models Over The Internetcited by 42
  5. 2024COAT: Compressing Optimizer states and Activation for Memory-Efficient FP8 Trainingcited by 32
  6. 2024How Far Are We From AGI: Are LLMs All We Need?cited by 31
  7. 2023FusionAI: Decentralized Training and Deploying LLMs with Massive Consumer-Level GPUscited by 20
  8. 2022lo-fi: distributed fine-tuning without communicationcited by 19
  9. 2023Near-Optimal Nonconvex-Strongly-Convex Bilevel Optimization with Fully First-Order Oraclescited by 16
  10. 2023Quantized Distributed Training of Large Models with Convergence Guaranteescited by 13
  11. 2025Parallax: Efficient LLM Inference Service over Decentralized Environmentcited by 12
  12. 2023Does compressing activations help model parallel training?cited by 11
17 more
  1. 2025Low-Precision Training of Large Language Models: Methods, Challenges, and Opportunitiescited by 11
  2. 2023Hazards from Increasingly Accessible Fine-Tuning of Downloadable Foundation Modelscited by 7
  3. 2024FusionLLM: A Decentralized LLM Training System on Geo-distributed GPUs with Adaptive Compressioncited by 7
  4. 2025CR-Net: Scaling Parameter-Efficient Training with Cross-Layer Low-Rank Structurecited by 7
  5. 2024Activations and Gradients Compression for Model-Parallel Trainingcited by 6
  6. 2025MergeQuant: Accurate 4-bit Static Quantization of Large Language Models by Channel-wise Calibrationcited by 3
  7. 2025TAH-QUANT: Effective Activation Quantization in Pipeline Parallelism over Slow Networkcited by 2
  8. 2025Towards Decentralized and Sustainable Foundation Model Training with the Edgecited by 2
  9. 2022RCD-SGD: Resource-Constrained Distributed SGD in Heterogeneous Environment via Submodular Partitioningcited by 1
  10. 2025Efficient AllReduce with Stragglerscited by 1
  11. 2025Partial Parameter Updates for Efficient Distributed Trainingcited by 1
  12. 2026TACO: Efficient Communication Compression of Intermediate Tensors for Scalable Tensor-Parallel LLM Trainingcited by 1
  13. 2025AMAQ: Adaptive Mixed-bit Activation Quantization for Collaborative Parameter Efficient Fine-tuningcited by 0
  14. 2026On the Convergence of Stochastic Gradient Descent with Perturbed Forward-Backward Passescited by 0
  15. 2026Learned Subspace Compression for Communication-Efficient Pipeline Parallelismcited by 0
  16. 2026SplitLite: Low-Rank Residual Compression for Split Learningcited by 0
  17. 2026Communication-Efficient LLM Adaptation over Decentralized GPU Meshescited by 0