mlsanityData to 14 Oct 2026

SDP4Bit: Toward 4-bit Communication Quantization in Sharded Data Parallelism for LLM Training

arXiv 2410.15526 ↗First posted 20 Oct 2024cs.LG cs.DCcited by 14

Not on the map. Why?

Abstract (arXiv)

Recent years have witnessed a clear trend towards language models with an ever-increasing number of parameters, as well as the growing training overhead and memory usage. Distributed training, particularly through Sharded Data Parallelism (ShardedDP) which partitions optimizer states among workers, has emerged as a crucial technique to mitigate training time and memory usage. Yet, a major challenge in the scalability of ShardedDP is the intensive communication of weights and gradients. While compression techniques can alleviate this issue, they often result in worse accuracy. Driven by this limitation, we propose SDP4Bit (Toward 4Bit Communication Quantization in Sharded Data Parallelism for LLM Training), which effectively reduces the communication of weights and gradients to nearly 4 bits via two novel techniques: quantization on weight differences, and two-level gradient smooth quantization. Furthermore, SDP4Bit presents an algorithm-system co-design with runtime optimization to minimize the computation overhead of compression. In addition to the theoretical guarantees of convergence, we empirically evaluate the accuracy of SDP4Bit on the pre-training of GPT models with up to 6.7 billion parameters, and the results demonstrate a negligible impact on training loss. Furthermore, speed experiments show that SDP4Bit achieves up to 4.08× speedup in end-to-end throughput on a scale of 128 GPUs.

Key facts. Cited by 14 papers in the map (data through 14 Oct 2026). Builds on 29 of its 38 references that are in the map. First posted on arXiv 20 Oct 2024.

Builds on

29 of 38 references.

References · 29

  1. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  2. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  3. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  4. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  5. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
  6. 2017Mixed Precision Trainingcited by 1,156
  7. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,121
  8. 2018Local SGD Converges Fast and Communicates Littlecited by 605
  9. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 526
  10. 2019Q-BERT: Hessian Based Ultra Low Precision Quantization of BERTcited by 359
  11. 2019Error Feedback Fixes SignSGD and other Gradient Compression Schemescited by 310
  12. 2019PowerSGD: Practical Low-Rank Gradient Compression for Distributed Optimizationcited by 238
17 more
  1. 2018Don't Use Large Mini-Batches, Use Local SGDcited by 202
  2. 2019Qsparse-local-SGD: Distributed SGD with Quantization, Sparsification, and Local Computationscited by 181
  3. 2020Is Local SGD Better than Minibatch SGD?cited by 151
  4. 2020Minibatch vs Local SGD for Heterogeneous Distributed Learningcited by 147
  5. 2018ATOMO: Communication-efficient Learning via Atomic Sparsificationcited by 136
  6. 2021EF21: A New, Simpler, Theoretically Better, and Practically Faster Error Feedbackcited by 133
  7. 2023Memory-Efficient Fine-Tuning of Compressed Large Language Models via sub-4-bit Integer Quantizationcited by 100
  8. 2019DoubleSqueeze: Parallel Stochastic Gradient Descent with Double-Pass Error-Compensated Compressioncited by 99
  9. 2023FP8-LM: Training FP8 Large Language Modelscited by 77
  10. 20211-bit Adam: Communication Efficient Large-Scale Training with Adam's Convergence Speedcited by 75
  11. 2023Training Transformers with 4-bit Integerscited by 60
  12. 2023ZeRO++: Extremely Efficient Collective Communication for Giant Model Trainingcited by 52
  13. 2020Adaptive Gradient Quantization for Data-Parallel SGDcited by 45
  14. 2019Asynchronous Decentralized SGD with Quantized and Local Updatescited by 36
  15. 2023THC: Accelerating Distributed Deep Learning Using Tensor Homomorphic Compressioncited by 15
  16. 2023Quantized Distributed Training of Large Models with Convergence Guaranteescited by 13
  17. 2023gZCCL: Compression-Accelerated Collective Communication Framework for GPU Clusterscited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 7; 2026: 7.

14 papers.

Sort:

By area:

  1. 2025Identifying Sensitive Weights via Post-quantization Integralcited by 11
  2. 2025Low-Precision Training of Large Language Models: Methods, Challenges, and Opportunitiescited by 11
  3. 2025Low-bit Model Quantization for Deep Neural Networks: A Surveycited by 11
  4. 2025Greedy Low-Rank Gradient Compression for Distributed Learning with Convergence Guaranteescited by 11
  5. 2026ENEC: A Lossless AI Model Compression Method Enabling Fast Inference on Ascend NPUscited by 3
  6. 2025EQuARX: Efficient Quantized AllReduce in XLA for Distributed Machine Learning Accelerationcited by 2
  7. 2026TACO: Efficient Communication Compression of Intermediate Tensors for Scalable Tensor-Parallel LLM Trainingcited by 2
  8. 2025Distributed Training under Packet Losscited by 1
  9. 2025Pareto-optimal Trade-offs Between Communication and Computation with Flexible Gradient Trackingcited by 0
  10. 2026GNMR: Runtime Stability Control for Low-Precision Large Language Model Trainingcited by 0
  11. 2026SCAPE: Accurate and Efficient LLM Training with Extreme Sparse Communicationcited by 0
  12. 2026GeoFP8: Geometry-Aware FP8 Gradient Compression for Distributed LLM Trainingcited by 0
2 more
  1. 2026CIERA: Cross-Iteration Exponent Reuse for Lossless Allgather in Sharded MoE Trainingcited by 0
  2. 2026Echo: Error-Compensated 4-Bit Tensor-Parallel Communication for LLM Training and Inferencecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026ScaleBITS: Scalable Bitwidth Search for Hardware-Aligned Mixed-Precision LLMs
  2. 2025FP4 All the Way: Fully Quantized Training of LLMs
  3. 2024LLMEasyQuant: Scalable Quantization for Parallel and Distributed LLM Inference
  4. 2025Opt4GPTQ: Co-Optimizing Memory and Computation for 4-bit GPTQ Quantized LLM Inference on Heterogeneous Platforms
  5. 2024Optimizing Large Language Models through Quantization: A Comparative Analysis of PTQ and QAT Techniques
  6. 2024Towards Low-bit Communication for Tensor Parallel LLM Inference
  7. 2025Rethinking 1-bit Optimization Leveraging Pre-trained Large Language Models
  8. 2026FOCUS: FP4 Optimization via Coupled-Relaxation and Dual-Granularity Scaling