mlsanityData to 7 Oct 2026

Memory Efficient Optimizers with 4-bit States

arXiv 2309.01507 ↗First posted 4 Sep 2023cs.LG cs.AIcited by 54

Not on the map. Why?

Abstract (arXiv)

Optimizer states are a major source of memory consumption for training neural networks, limiting the maximum trainable model within given memory budget. Compressing the optimizer states from 32-bit floating points to lower bitwidth is promising to reduce the training memory footprint, while the current lowest achievable bitwidth is 8-bit. In this work, we push optimizer states bitwidth down to 4-bit through a detailed empirical analysis of first and second moments. Specifically, we find that moments have complicated outlier patterns, that current block-wise quantization cannot accurately approximate. We use a smaller block size and propose to utilize both row-wise and column-wise information for better quantization. We further identify a zero point problem of quantizing the second moment, and solve this problem with a linear quantizer that excludes the zero point. Our 4-bit optimizers are evaluated on a wide variety of benchmarks including natural language understanding, machine translation, image classification, and instruction tuning. On all the tasks our optimizers can achieve comparable accuracy with their full-precision counterparts, while enjoying better memory efficiency.

Key facts. Cited by 54 papers in the map (data through 7 Oct 2026). Builds on 44 of its 58 references that are in the map. First posted on arXiv 4 Sep 2023.

Builds on

44 of 58 references.

References · 44

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  7. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  8. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  9. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  10. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  11. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  12. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
32 more
  1. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  2. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  3. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  4. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  5. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  6. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  7. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  8. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  9. 2022Emergent Abilities of Large Language Modelscited by 2,483
  10. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  11. 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,548
  12. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,538
  13. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  14. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
  15. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 896
  16. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 881
  17. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 837
  18. 2016Training Deep Nets with Sublinear Memory Costcited by 825
  19. 2015BinaryConnect: Training Deep Neural Networks with binary weights during propagationscited by 571
  20. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 517
  21. 20218-bit Optimizers via Block-wise Quantizationcited by 395
  22. 2021ZeRO-Offload: Democratizing Billion-Scale Model Trainingcited by 339
  23. 2021ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learningcited by 253
  24. 2017The E2E Dataset: New Challenges For End-to-End Generationcited by 252
  25. 2022The case for 4-bit precision: k-bit Inference Scaling Lawscited by 247
  26. 2022Delta Tuning: A Comprehensive Study of Parameter Efficient Methods for Pre-trained Language Modelscited by 179
  27. 2022Outlier Suppression: Pushing the Limit of Low-bit Transformer Language Modelscited by 176
  28. 2021Understanding and Overcoming the Challenges of Efficient Transformer Quantizationcited by 139
  29. 2021ActNN: Reducing Training Memory Footprint via 2-Bit Activation Compressed Trainingcited by 61
  30. 20158-Bit Approximations for Parallelism in Deep Learningcited by 56
  31. 2019Memory-Efficient Adaptive Optimizationcited by 19
  32. 2019Extreme Tensoring for Low-Memory Preconditioningcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 21; 2025: 21; 2026: 11.

54 papers.

Sort:

By area:

  1. 2024GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projectioncited by 447
  2. 2024SageAttention: Accurate 8-Bit Attention for Plug-and-play Inference Accelerationcited by 158
  3. 2024Adam-mini: Use Fewer Learning Rates To Gain Morecited by 134
  4. 2025In Search of Adam's Secret Saucecited by 40
  5. 2024COAT: Compressing Optimizer states and Activation for Memory-Efficient FP8 Trainingcited by 32
  6. 2024A Survey of Low-bit Large Language Models: Basics, Systems, and Algorithmscited by 30
  7. 2024Zeroth-Order Fine-Tuning of LLMs in Random Subspacescited by 23
  8. 2024VeLoRA: Memory Efficient Training using Rank-1 Sub-Token Projectionscited by 18
  9. 2024Grass: Compute Efficient Low-Memory LLM Training with Structured Sparse Gradientscited by 17
  10. 20244-bit Shampoo for Memory-Efficient Network Trainingcited by 16
  11. 2024AdaRankGrad: Adaptive Gradient-Rank and Moments for Memory-Efficient LLMs Training and Fine-Tuningcited by 16
  12. 2024Propulsion: Steering LLM with Tiny Fine-Tuningcited by 14
38 more
  1. 2025SubTrack++ : Gradient Subspace Tracking for Scalable LLM Trainingcited by 14
  2. 2024Memory-Efficient Optimization with Factorized Hamiltonian Descentcited by 11
  3. 2025Low-Precision Training of Large Language Models: Methods, Challenges, and Opportunitiescited by 11
  4. 2023QFT: Quantized Full-parameter Tuning of LLMs with Affordable Resourcescited by 10
  5. 2024Exploring Quantization for Efficient Pre-Training of Transformer Language Modelscited by 10
  6. 2024Parameter-Efficient Fine-Tuning of Large Language Models using Semantic Knowledge Tuningcited by 10
  7. 2025Identifying Sensitive Weights via Post-quantization Integralcited by 10
  8. 2025Leveraging Coordinate Momentum in SignSGD and Muon: Memory-Optimized Zero-Ordercited by 9
  9. 2025SAGE: A Framework of Precise Retrieval for RAGcited by 8
  10. 2024Adapprox: Adaptive Approximation in Adam Optimization via Randomized Low-Rank Matricescited by 7
  11. 2024FRUGAL: Memory-Efficient Optimization by Reducing State Overhead for Scalable Trainingcited by 7
  12. 2026Full-Stack FP4: Stable LLM Pretraining with Quantized Projections, Optimizers, and Attentioncited by 7
  13. 2024100K or 100 Days: Trade-offs when Pre-Training with Academic Resourcescited by 6
  14. 2024No More Adam: Learning Rate Scaling at Initialization is All You Needcited by 6
  15. 2025Breaking the Frozen Subspace: Importance Sampling for Low-Rank Optimization in LLM Pretrainingcited by 6
  16. 2025GradPower: Powering Gradients for Faster Language Model Pre-Trainingcited by 6
  17. 2026Understanding Quantization of Optimizer States in LLM Pre-training: Dynamics of State Staleness and Effectiveness of State Resetscited by 6
  18. 2025Pushing the Limits of Low-Bit Optimizers: A Focus on EMA Dynamicscited by 5
  19. 2025Low-rank Momentum Factorization for Memory Efficient Trainingcited by 4
  20. 2026FlashOptim: Optimizers for Memory-Efficient Trainingcited by 4
  21. 2024Sketch to Adapt: Fine-Tunable Sketches for Efficient LLM Adaptationcited by 3
  22. 2025LORENZA: Enhancing Generalization in Low-Rank Gradient LLM Training via Efficient Zeroth-Order Adaptive SAMcited by 3
  23. 2026ECO: Quantized Training without Full-Precision Master Weightscited by 3
  24. 2024Memory-Efficient 4-bit Preconditioned Stochastic Optimizationcited by 2
  25. 2025MISA: Memory-Efficient LLMs Optimization with Module-wise Importance Samplingcited by 2
  26. 2024Lean and Mean Adaptive Optimization via Subset-Norm and Subspace-Momentum with Convergence Guaranteescited by 1
  27. 2025On the Duality between Gradient Transformations and Adapterscited by 1
  28. 2025MLorc: Momentum Low-rank Compression for Memory Efficient Large Language Model Adaptationcited by 1
  29. 2025Partial Parameter Updates for Efficient Distributed Trainingcited by 1
  30. 2026Gefen: Optimized Stochastic Optimizercited by 1
  31. 2025Irrational Complex Rotations Empower Low-bit Optimizerscited by 0
  32. 2025Beyond the model: Key differentiators in large language models and multi-agent servicescited by 0
  33. 2025Reducing Storage of Pretrained Neural Networks by Rate-Constrained Quantization and Entropy Codingcited by 0
  34. 2025EMLoC: Emulator-based Memory-efficient Fine-tuning with LoRA Correctioncited by 0
  35. 2025BitSnap: Checkpoint Sparsification and Quantization in LLM Trainingcited by 0
  36. 2025Alada: Alternating Adaptation of Momentum Method for Memory-Efficient Matrix Optimizationcited by 0
  37. 2026Zero-Order Optimization for LLM Fine-Tuning via Learnable Direction Samplingcited by 0
  38. 2026AGoQ: Activation and Gradient Quantization for Memory-Efficient Distributed Training of LLMscited by 0

50 most cited of 54 All 54 (JSON)