mlsanityData to 14 Oct 2026

EDGC: Entropy-driven Dynamic Gradient Compression for Efficient LLM Training

arXiv 2511.10333 ↗First posted 13 Nov 2025cs.LG cs.AI cs.PFcited by 2

Not on the map. Why?

Abstract (arXiv)

Training large language models (LLMs) at scale incurs substantial communication overhead, while static gradient compression cannot adapt to gradient evolution and may degrade model quality. We propose EDGC, an entropy-driven dynamic gradient compression framework that adapts compression ranks to gradient entropy during training. EDGC combines efficient entropy estimation through gradient sampling, a theoretical model relating entropy to compression rank under a bounded-error constraint, and window-based rank adjustment across pipeline stages. Experiments on 32-V100 and 64-H100 GPU clusters training GPT2 models with 2.5B and 12.1B parameters show that EDGC reduces communication latency by up to 46.45% and end-to-end training time by 16.13%, while maintaining model quality.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 14 of its 32 references that are in the map. First posted on arXiv 13 Nov 2025.

Builds on

14 of 32 references.

References · 14

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,045
  3. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  4. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  5. 2021Carbon Emissions and Large Neural Network Trainingcited by 567
  6. 2018Characterizing Implicit Bias in Terms of Optimization Geometrycited by 266
  7. 2019PowerSGD: Practical Low-Rank Gradient Compression for Distributed Optimizationcited by 238
  8. 20211-bit Adam: Communication Efficient Large-Scale Training with Adam's Convergence Speedcited by 75
  9. 2017Poseidon: An Efficient Communication Architecture for Distributed Deep Learning on GPU Clusterscited by 57
  10. 20158-Bit Approximations for Parallelism in Deep Learningcited by 56
  11. 2021Distribution Adaptive INT8 Quantization for Training CNNscited by 28
  12. 2021ScaleCom: Scalable Sparsified Gradient Compression for Communication-Efficient Distributed Trainingcited by 22
2 more
  1. 2023Optimus-CC: Efficient Large NLP Model Training with 3D Parallelism Aware Communication Compressioncited by 15
  2. 2019MG-WFBP: Merging Gradients Wisely for Efficient Communication in Distributed Deep Learningcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026TACO: Efficient Communication Compression of Intermediate Tensors for Scalable Tensor-Parallel LLM Trainingcited by 2
  2. 2026SCAPE: Accurate and Efficient LLM Training with Extreme Sparse Communicationcited by 0