mlsanityData to 14 Oct 2026

LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale

arXiv 2208.07339 ↗First posted 15 Aug 2022cs.LG cs.AIcited by 906

Not on the map. Why?

Cited by 94 in Efficient LLM inference and serving, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Large language models have been widely adopted but require significant GPU memory for inference. We develop a procedure for Int8 matrix multiplication for feed-forward and attention projection layers in transformers, which cut the memory needed for inference by half while retaining full precision performance. With our method, a 175B parameter 16/32-bit checkpoint can be loaded, converted to Int8, and used immediately without performance degradation. This is made possible by understanding and working around properties of highly systematic emergent features in transformer language models that dominate attention and transformer predictive performance. To cope with these features, we develop a two-part quantization procedure, LLM.int8(). We first use vector-wise quantization with separate normalization constants for each inner product in the matrix multiplication, to quantize most of the features. However, for the emergent outliers, we also include a new mixed-precision decomposition scheme, which isolates the outlier feature dimensions into a 16-bit matrix multiplication while still more than 99.9% of values are multiplied in 8-bit. Using LLM.int8(), we show empirically it is possible to perform inference in LLMs with up to 175B parameters without any performance degradation. This result makes such models much more accessible, for example making it possible to use OPT-175B/BLOOM on a single server with consumer GPUs. We open-source our software.

Key facts. Cited by 906 papers in the map (data through 14 Oct 2026). Builds on 48 of its 82 references that are in the map. First posted on arXiv 15 Aug 2022.

Builds on

48 of 82 references.

References · 48

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  6. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  7. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  8. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,218
  9. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,488
  10. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,043
  11. 2016XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networkscited by 849
  12. 2021A Survey of Quantization Methods for Efficient Neural Network Inferencecited by 815
36 more
  1. 2019Learned Step Size Quantizationcited by 628
  2. 2015BinaryConnect: Training Deep Neural Networks with binary weights during propagationscited by 592
  3. 2019CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Datacited by 548
  4. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 526
  5. 2020Scaling Laws for Autoregressive Generative Modelingcited by 522
  6. 20218-bit Optimizers via Block-wise Quantizationcited by 403
  7. 2019HAWQ: Hessian AWare Quantization of Neural Networks with Mixed-Precisioncited by 402
  8. 2019Q-BERT: Hessian Based Ultra Low Precision Quantization of BERTcited by 359
  9. 2019Q8BERT: Quantized 8Bit BERTcited by 300
  10. 2018LQ-Nets: Learned Quantization for Highly Accurate and Compact Deep Neural Networkscited by 274
  11. 2019Representation Degeneration Problem in Training Natural Language Generation Modelscited by 255
  12. 2019Differentiable Soft Quantization: Bridging Full-Precision and Low-Bit Neural Networkscited by 238
  13. 2018A Simple Method for Commonsense Reasoningcited by 223
  14. 2016Trained Ternary Quantizationcited by 208
  15. 2018Scaling Neural Machine Translationcited by 204
  16. 2020Binary Neural Networks: A Surveycited by 190
  17. 2020Integer Quantization for Deep Learning Inference: Principles and Empirical Evaluationcited by 188
  18. 2020BinaryBERT: Pushing the Limit of BERT Quantizationcited by 178
  19. 2021Efficient Large Scale Language Modeling with Mixtures of Expertscited by 175
  20. 2018Mesh-TensorFlow: Deep Learning for Supercomputerscited by 169
  21. 2018Training Deep Neural Networks with 8-bit Floating Point Numberscited by 164
  22. 2021All Bark and No Bite: Rogue Dimensions in Transformer Language Models Obscure Representational Qualitycited by 156
  23. 2020TernaryBERT: Distillation-aware Ultra-low Bit BERTcited by 148
  24. 2020Training with Quantization Noise for Extreme Model Compressioncited by 145
  25. 2021Understanding and Overcoming the Challenges of Efficient Transformer Quantizationcited by 141
  26. 2021BERT Busters: Outlier Dimensions that Disrupt Transformerscited by 112
  27. 2016Edinburgh Neural Machine Translation Systems for WMT 16cited by 76
  28. 2022Outliers Dimensions that Disrupt Transformers Are Driven by Frequencycited by 55
  29. 2020A Statistical Framework for Low-bitwidth Training of Deep Neural Networkscited by 51
  30. 2021FBGEMM: Enabling High-Performance Low-Precision Deep Learning Inferencecited by 41
  31. 2020Towards Fully 8-bit Integer Inference for the Transformer Modelcited by 35
  32. 2020Positional Artefacts Propagate Through Masked Language Model Embeddingscited by 35
  33. 2022F8Net: Fixed-Point 8-bit Only Multiplication for Network Quantizationcited by 28
  34. 2019Mixed Precision Training With 8-bit Floating Pointcited by 23
  35. 2020Shifted and Squeezed 8-bit Floating Point format for Low-Precision Training of Deep Neural Networkscited by 23
  36. 2021Automatic Mixed-Precision Quantization Search of BERTcited by 16

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 17; 2023: 203; 2024: 222; 2025: 169; 2026: 295.

906 papers.

Sort:

By area:

  1. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  2. 2023Qwen Technical Reportcited by 3,344
  3. 2023A Survey of Large Language Modelscited by 2,576
  4. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
  5. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  6. 2023Sparse Autoencoders Find Highly Interpretable Features in Language Modelscited by 1,420
  7. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
  8. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
  9. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,298
  10. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,173
  11. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,121
  12. 2023Accelerating Large Language Model Decoding with Speculative Samplingcited by 1,006
38 more
  1. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 843
  2. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Headscited by 843
  3. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 832
  4. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 800
  5. 2024Chronos: Learning the Language of Time Seriescited by 759
  6. 2024Yi: Open Foundation Models by 01.AIcited by 719
  7. 2023LLM-Pruner: On the Structural Pruning of Large Language Modelscited by 650
  8. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 637
  9. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 591
  10. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 575
  11. 2022Efficiently Scaling Transformer Inferencecited by 539
  12. 2024SpinQuant: LLM quantization with learned rotationscited by 471
  13. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Timecited by 439
  14. 2023OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Modelscited by 424
  15. 2023Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruningcited by 415
  16. 2023QuIP: 2-Bit Quantization of Large Language Models With Guaranteescited by 414
  17. 2023How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resourcescited by 408
  18. 2024SliceGPT: Compress Large Language Models by Deleting Rows and Columnscited by 385
  19. 2024ShortGPT: Layers in Large Language Models are More Redundant Than You Expectcited by 383
  20. 2023SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compressioncited by 367
  21. 2023SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verificationcited by 365
  22. 2023LLM-QAT: Data-Free Quantization Aware Training for Large Language Modelscited by 324
  23. 2023CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval-Xcited by 300
  24. 2024Aya Model: An Instruction Finetuned Open-Access Multilingual Language Modelcited by 295
  25. 2023SqueezeLLM: Dense-and-Sparse Quantizationcited by 294
  26. 2024Massive Activations in Large Language Modelscited by 294
  27. 2023LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Modelscited by 287
  28. 2023MedAlpaca -- An Open-Source Collection of Medical Conversational AI Models and Training Datacited by 286
  29. 2024Do Llamas Work in English? On the Latent Language of Multilingual Transformerscited by 281
  30. 2023Q-Diffusion: Quantizing Diffusion Modelscited by 273
  31. 2023A Survey on Model Compression for Large Language Modelscited by 269
  32. 2023Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Timecited by 252
  33. 2022The case for 4-bit precision: k-bit Inference Scaling Lawscited by 251
  34. 2024Extreme Compression of Large Language Models via Additive Quantizationcited by 232
  35. 2024The Unreasonable Ineffectiveness of the Deeper Layerscited by 226
  36. 2024MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Casescited by 223
  37. 2024A Survey on Efficient Inference for Large Language Modelscited by 205
  38. 2023CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Servingcited by 203

50 most cited of 906 All 906 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024FrameQuant: Flexible Low-Bit Quantization for Transformers
  2. 2026BAPS: A Fine-Grained Low-Precision Scheme for Softmax in Attention via Block-Aware Precision reScaling
  3. 2023Training Transformers with 4-bit Integers
  4. 2025TWEO: Transformers Without Extreme Outliers Enables FP8 Training And Quantization For Dummies
  5. 2026The Quantization Benefits of Residual-Free Transformers
  6. 2023Quantization Variation: A New Perspective on Training Transformers with Low-Bit Precision
  7. 2024SageAttention: Accurate 8-Bit Attention for Plug-and-play Inference Acceleration
  8. 2026GoQuant: Geometric Orthogonal Residual Projection for Multiplier-Free Power-of-Two Transformer Quantization