mlsanityData to 7 Oct 2026

8-bit Optimizers via Block-wise Quantization

arXiv 2110.02861 ↗First posted 6 Oct 2021cs.LGcited by 395

Not on the map. Why?

Abstract (arXiv)

Stateful optimizers maintain gradient statistics over time, e.g., the exponentially smoothed sum (SGD with momentum) or squared sum (Adam) of past gradient values. This state can be used to accelerate optimization compared to plain stochastic gradient descent but uses memory that might otherwise be allocated to model parameters, thereby limiting the maximum size of models trained in practice. In this paper, we develop the first optimizers that use 8-bit statistics while maintaining the performance levels of using 32-bit optimizer states. To overcome the resulting computational, quantization, and stability challenges, we develop block-wise dynamic quantization. Block-wise quantization divides input tensors into smaller blocks that are independently quantized. Each block is processed in parallel across cores, yielding faster optimization and high precision quantization. To maintain stability and performance, we combine block-wise quantization with two additional changes: (1) dynamic quantization, a form of non-linear optimization that is precise for both large and small magnitude values, and (2) a stable embedding layer to reduce gradient variance that comes from the highly non-uniform distribution of input tokens in language models. As a result, our 8-bit optimizers maintain 32-bit performance with a small fraction of the memory footprint on a range of tasks, including 1.5B parameter language modeling, GLUE finetuning, ImageNet classification, WMT'14 machine translation, MoCo v2 contrastive ImageNet pretraining+finetuning, and RoBERTa pretraining, without changes to the original optimizer hyperparameters. We open-source our 8-bit optimizers as a drop-in replacement that only requires a two-line code change.

Key facts. Cited by 395 papers in the map (data through 7 Oct 2026). Builds on 47 of its 70 references that are in the map. First posted on arXiv 6 Oct 2021.

Builds on

47 of 70 references.

References · 47

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  5. 2020Language Models are Few-Shot Learnerscited by 32,199
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  7. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  8. 2020Scaling Laws for Neural Language Modelscited by 6,411
  9. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  10. 2016Layer Normalizationcited by 5,579
  11. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  12. 2021Zero-Shot Text-to-Image Generationcited by 4,276
35 more
  1. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  2. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  3. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,282
  4. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  5. 2017Decoupled Weight Decay Regularizationcited by 1,722
  6. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  7. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  8. 2017Mixed Precision Trainingcited by 1,135
  9. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,041
  10. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 896
  11. 2016XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networkscited by 829
  12. 2016Training Deep Nets with Sublinear Memory Costcited by 825
  13. 2019CTRL: A Conditional Transformer Language Model for Controllable Generationcited by 754
  14. 2015BinaryConnect: Training Deep Neural Networks with binary weights during propagationscited by 571
  15. 2019CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Datacited by 544
  16. 2020Scaling Laws for Autoregressive Generative Modelingcited by 510
  17. 2021BASE Layers: Simplifying Training of Large, Sparse Modelscited by 315
  18. 2021ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learningcited by 253
  19. 2019Differentiable Soft Quantization: Bridging Full-Precision and Low-Bit Neural Networkscited by 234
  20. 2018A Simple Method for Commonsense Reasoningcited by 223
  21. 2017The Reversible Residual Network: Backpropagation Without Storing Activationscited by 215
  22. 2018Scaling Neural Machine Translationcited by 204
  23. 2016Trained Ternary Quantizationcited by 203
  24. 2020Binary Neural Networks: A Surveycited by 184
  25. 2018Training Deep Neural Networks with 8-bit Floating Point Numberscited by 163
  26. 2018PipeDream: Fast and Efficient Pipeline Parallel DNN Trainingcited by 116
  27. 2016Edinburgh Neural Machine Translation Systems for WMT 16cited by 76
  28. 20158-Bit Approximations for Parallelism in Deep Learningcited by 56
  29. 2018Training DNNs with Hybrid Block Floating Pointcited by 39
  30. 2020Training Large Neural Networks with Constant Memory using a New Execution Algorithmcited by 34
  31. 2020Shifted and Squeezed 8-bit Floating Point format for Low-Precision Training of Deep Neural Networkscited by 23
  32. 2019Mixed Precision Training With 8-bit Floating Pointcited by 22
  33. 2019Memory-Efficient Adaptive Optimizationcited by 19
  34. 2019Computing Extremely Accurate Quantiles Using t-Digestscited by 10
  35. 2019Extreme Tensoring for Low-Memory Preconditioningcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 2; 2022: 20; 2023: 57; 2024: 103; 2025: 114; 2026: 99.

395 papers.

Sort:

By area:

  1. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  2. 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,442
  3. 2023BloombergGPT: A Large Language Model for Financecited by 923
  4. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 881
  5. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 837
  6. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Headscited by 821
  7. 2021MetaICL: Learning to Learn In Contextcited by 450
  8. 2024GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projectioncited by 447
  9. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Modelscited by 391
  10. 2023Fine-Tuning Language Models with Just Forward Passescited by 337
  11. 2023MedAlpaca -- An Open-Source Collection of Medical Conversational AI Models and Training Datacited by 282
  12. 2022The case for 4-bit precision: k-bit Inference Scaling Lawscited by 247
38 more
  1. 2023JourneyDB: A Benchmark for Generative Image Understandingcited by 203
  2. 2024SOAP: Improving and Stabilizing Shampoo using Adamcited by 179
  3. 2025SANA 1.5: Efficient Scaling of Training-Time and Inference-Time Compute in Linear Diffusion Transformercited by 165
  4. 2022Separable Self-attention for Mobile Vision Transformerscited by 163
  5. 2023A Survey of Large Language Models for Healthcare: from Data, Technology, and Applications to Accountability and Ethicscited by 162
  6. 2024RVT-2: Learning Precise Manipulation from Few Demonstrationscited by 135
  7. 2023Full Parameter Fine-tuning for Large Language Models with Limited Resourcescited by 134
  8. 2024Adam-mini: Use Fewer Learning Rates To Gain Morecited by 134
  9. 2023ChatEDA: A Large Language Model Powered Autonomous Agent for EDAcited by 118
  10. 2024PLeak: Prompt Leaking Attacks against Large Language Model Applicationscited by 113
  11. 2023Navigating Text-To-Image Customization: From LyCORIS Fine-Tuning to Model Evaluationcited by 108
  12. 2024Flora: Low-Rank Adapters Are Secretly Gradient Compressorscited by 103
  13. 2022What Language Model to Train if You Have One Million GPU Hours?cited by 92
  14. 2024Fine-Tuned Language Models Generate Stable Inorganic Materials as Textcited by 92
  15. 2024Scaling Laws for Precisioncited by 92
  16. 2023Agent Attention: On the Integration of Softmax and Linear Attentioncited by 90
  17. 2022Petals: Collaborative Inference and Fine-tuning of Large Modelscited by 84
  18. 2023Sensitivity, Performance, Robustness: Deconstructing the Effect of Sociodemographic Promptingcited by 80
  19. 2022Efficient Methods for Natural Language Processing: A Surveycited by 78
  20. 2023Graph-ToolFormer: To Empower LLMs with Graph Reasoning Ability via Prompt Augmented by ChatGPTcited by 78
  21. 2023FP8-LM: Training FP8 Large Language Modelscited by 75
  22. 2023EQ-Bench: An Emotional Intelligence Benchmark for Large Language Modelscited by 75
  23. 2023Stable and low-precision training for large-scale vision-language modelscited by 74
  24. 2023Advancing Transformer Architecture in Long-Context Large Language Models: A Comprehensive Surveycited by 71
  25. 2023Attention Satisfies: A Constraint-Satisfaction Lens on Factual Errors of Language Modelscited by 67
  26. 2024Model Compression and Efficient Inference for Large Language Models: A Surveycited by 62
  27. 2022Eco2AI: carbon emissions tracking of machine learning models as the first step towards sustainable AIcited by 58
  28. 2023SWARM Parallelism: Training Large Models Can Be Surprisingly Communication-Efficientcited by 58
  29. 2023Memory Efficient Optimizers with 4-bit Statescited by 54
  30. 2023Spike No More: Stabilizing the Pre-training of Large Language Modelscited by 51
  31. 2023ZeRO++: Extremely Efficient Collective Communication for Giant Model Trainingcited by 50
  32. 2024Variance-reduced Zeroth-Order Methods for Fine-Tuning Language Modelscited by 50
  33. 2024Automated Federated Pipeline for Parameter-Efficient Fine-Tuning of Large Language Modelscited by 49
  34. 2022HyperTuning: Toward Adapting Large Language Models without Back-propagationcited by 45
  35. 2024Margin-aware Preference Optimization for Aligning Diffusion Models without Referencecited by 45
  36. 2023PiVe: Prompting with Iterative Verification Improving Graph-based Generative Capability of LLMscited by 44
  37. 2024SLTrain: a sparse plus low-rank approach for parameter and memory efficient pretrainingcited by 44
  38. 2023KCTS: Knowledge-Constrained Tree Search Decoding with Token-Level Hallucination Detectioncited by 43

50 most cited of 395 All 395 (JSON)