mlsanityData to 7 Oct 2026

Adaptive Gradient Quantization for Data-Parallel SGD

arXiv 2010.12460 ↗First posted 23 Oct 2020cs.LG stat.MLcited by 43

Not on the map. Why?

Abstract (arXiv)

Many communication-efficient variants of SGD use gradient quantization schemes. These schemes are often heuristic and fixed over the course of training. We empirically observe that the statistics of gradients of deep models change during the training. Motivated by this observation, we introduce two adaptive quantization schemes, ALQ and AMQ. In both schemes, processors update their compression schemes in parallel by efficiently computing sufficient statistics of a parametric distribution. We improve the validation accuracy by almost 2% on CIFAR-10 and 1% on ImageNet in challenging low-cost communication setups. Our adaptive methods are also significantly more robust to the choice of hyperparameters.

Key facts. Cited by 43 papers in the map (data through 7 Oct 2026). First posted on arXiv 23 Oct 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 12; 2022: 7; 2023: 11; 2024: 5; 2025: 5; 2026: 3.

43 papers.

Sort:

  1. 2021A Survey of Quantization Methods for Efficient Neural Network Inferencecited by 798
  2. 2021NeRV: Neural Representations for Videoscited by 255
  3. 2021What Do We Mean by Generalization in Federated Learning?cited by 62
  4. 2021Adaptive Quantization of Model Updates for Communication-Efficient Federated Learningcited by 60
  5. 2021Edge-Cloud Polarization and Collaboration: A Comprehensive Survey for AIcited by 38
  6. 2024Lightweight Deep Learning for Resource-Constrained Environments: A Surveycited by 38
  7. 2021Moshpit SGD: Communication-Efficient Decentralized Training on Heterogeneous Unreliable Devicescited by 36
  8. 2022Fine-tuning Language Models over Slow Networks using Activation Compression with Guaranteescited by 29
  9. 2021Lossless Compression of Efficient Private Local Randomizerscited by 27
  10. 2024Differentiable Weightless Neural Networkscited by 27
  11. 2022lo-fi: distributed fine-tuning without communicationcited by 19
  12. 2022Adaptive Compression for Communication-Efficient Distributed Trainingcited by 14
31 more
  1. 2023Optimus-CC: Efficient Large NLP Model Training with 3D Parallelism Aware Communication Compressioncited by 14
  2. 2023Fast Optimal Locally Private Mean Estimation via Random Projectionscited by 14
  3. 2023Quantized Distributed Training of Large Models with Convergence Guaranteescited by 13
  4. 2024SDP4Bit: Toward 4-bit Communication Quantization in Sharded Data Parallelism for LLM Trainingcited by 13
  5. 2023Communication Compression for Byzantine Robust Learning: New Efficient Algorithms and Improved Ratescited by 12
  6. 2025Low-Precision Training of Large Language Models: Methods, Challenges, and Opportunitiescited by 11
  7. 2021Theoretically Better and Numerically Faster Distributed Optimization with Smoothness-Aware Quantization Techniquescited by 9
  8. 2024Communication-Efficient Large-Scale Distributed Deep Learning: A Comprehensive Surveycited by 9
  9. 2023Federated Learning under Covariate Shifts with Generalization Guaranteescited by 6
  10. 2021Distributed Online Learning for Joint Regret with Communication Constraintscited by 5
  11. 2022MixTailor: Mixed Gradient Aggregation for Robust Learning Against Tailored Attackscited by 5
  12. 2021CGX: Adaptive System Support for Communication-Efficient Deep Learningcited by 4
  13. 2023FedREP: A Byzantine-Robust, Communication-Efficient and Privacy-Preserving Framework for Federated Learningcited by 3
  14. 2021DeepReduce: A Sparse-tensor Communication Framework for Distributed Deep Learningcited by 2
  15. 2021Fundamental limits of over-the-air optimization: Are analog schemes optimal?cited by 2
  16. 2023Inshrinkerator: Compressing Deep Learning Training Checkpoints via Dynamic Quantizationcited by 2
  17. 2023Self-Distilled Quantization: Achieving High Compression Rates in Transformer-Based Language Modelscited by 2
  18. 2023Distributed Extra-gradient with Optimal Complexity and Communication Guaranteescited by 2
  19. 2024Optimal and Near-Optimal Adaptive Vector Quantizationcited by 2
  20. 2025On the Gradient Complexity of Private Optimization with Private Oraclescited by 2
  21. 2022Empirical Analysis on Top-k Gradient Sparsification for Distributed Deep Learning in a Supercomputing Environmentcited by 1
  22. 2022L-GreCo: Layerwise-Adaptive Gradient Compression for Efficient and Accurate Deep Learningcited by 1
  23. 2023CORE: Common Random Reconstruction for Distributed Optimization with Provable Low Communication Complexitycited by 1
  24. 2025Addressing Label Shift in Distributed Learning via Entropy Regularizationcited by 1
  25. 2025Layer-wise Quantization for Quantized Optimistic Dual Averagingcited by 1
  26. 2026LoKA: Low-precision Kernel Applications for Recommendation Models At Scalecited by 1
  27. 2026Entropy-Constrained Adaptive Stochastic Quantizationcited by 1
  28. 2022Variance Reduction is an Antidote to Byzantines: Better Rates, Weaker Assumptions and Communication Compression as a Cherry on the Topcited by 0
  29. 2023Contractive error feedback for gradient compressioncited by 0
  30. 2025It Takes a Good Model to Train a Good Model: Generalized Gaussian Priors for Optimized LLMscited by 0
  31. 2026Inner Product Aware Quantization: Provably Fast, Accurate, and Adaptive Algorithmscited by 0