mlsanityData to 7 Oct 2026

QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs

arXiv 2404.00456 ↗First posted 30 Mar 2024cs.LGcited by 598

On the map: Efficient LLM inference and serving → KV cache quantization

Abstract (arXiv)

We introduce QuaRot, a new Quantization scheme based on Rotations, which is able to quantize LLMs end-to-end, including all weights, activations, and KV cache in 4 bits. QuaRot rotates LLMs in a way that removes outliers from the hidden state without changing the output, making quantization easier. This computational invariance is applied to the hidden state (residual) of the LLM, as well as to the activations of the feed-forward components, aspects of the attention mechanism, and to the KV cache. The result is a quantized model where all matrix multiplications are performed in 4 bits, without any channels identified for retention in higher precision. Our 4-bit quantized LLaMa2-70B model has losses of at most 0.47 WikiText-2 perplexity and retains 99% of the zero-shot performance. We also show that QuaRot can provide lossless 6 and 8 bit LLaMa2 models without any calibration data using round-to-nearest quantization. Code is available at: https://github.com/spcl/QuaRot.

Key facts. Cited by 598 papers in the map (data through 7 Oct 2026). Placed in KV cache quantization, one of 12 directions in Efficient LLM inference and serving. Builds on 23 of its 33 references that are in the map. First posted on arXiv 30 Mar 2024.

Builds on

23 of 33 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache5 Feb
  2. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization31 Jan
  3. 2023Atom: Low-bit Quantization for Efficient and Accurate LLM Serving29 Oct

Other references · 20

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  2. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  3. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  4. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  5. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  6. 2016Pointer Sentinel Mixture Modelscited by 2,992
  7. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  8. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  9. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  10. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
  11. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  12. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 567
8 more
  1. 2023OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Modelscited by 403
  2. 2023QuIP: 2-Bit Quantization of Large Language Models With Guaranteescited by 397
  3. 2024SliceGPT: Compress Large Language Models by Deleting Rows and Columnscited by 378
  4. 2023SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compressioncited by 359
  5. 2024QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebookscited by 354
  6. 2024Extreme Compression of Large Language Models via Additive Quantizationcited by 223
  7. 2022Outlier Suppression: Pushing the Limit of Low-bit Transformer Language Modelscited by 176
  8. 2023Training Transformers with 4-bit Integerscited by 58

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 71; 2025: 229; 2026: 298.

598 papers.

Sort:

By area:

  1. 2026GLM-5: from Vibe Coding to Agentic Engineeringcited by 502
  2. 2024SpinQuant: LLM quantization with learned rotationscited by 446
  3. 2024QServe: W4A8KV4 Quantization and System Co-design for Efficient LLM Servingcited by 200
  4. 2024A Survey on Efficient Inference for Large Language Modelscited by 197
  5. 2024DuQuant: Distributing Outliers via Dual Transformation Makes Stronger Quantized LLMscited by 165
  6. 2024SVDQuant: Absorbing Outliers by Low-Rank Components for 4-Bit Diffusion Modelscited by 161
  7. 2024From Decoding to Meta-Generation: Inference-time Algorithms for Large Language Modelscited by 136
  8. 2024A Survey on Large Language Model Acceleration based on KV Cache Managementcited by 125
  9. 2024EfficientQAT: Efficient Quantization-Aware Training for Large Language Modelscited by 123
  10. 2024FlatQuant: Flatness Matters for LLM Quantizationcited by 116
  11. 2025TurboQuant: Online Vector Quantization with Near-optimal Distortion Ratecited by 116
  12. 2024SageAttention2: Efficient Attention with Thorough Outlier Smoothing and Per-thread INT4 Quantizationcited by 113
38 more
  1. 2024ViDiT-Q: Efficient and Accurate Quantization of Diffusion Transformers for Image and Video Generationcited by 110
  2. 2025OstQuant: Refining Large Language Model Quantization with Orthogonal and Scaling Transformations for Better Distribution Fittingcited by 94
  3. 2024QTIP: Quantization with Trellises and Incoherence Processingcited by 86
  4. 2025Quantization Hurts Reasoning? An Empirical Study on Quantized Reasoning Modelscited by 68
  5. 2025Pretraining Large Language Models with NVFP4cited by 67
  6. 2024Palu: Compressing KV-Cache with Low-Rank Projectioncited by 59
  7. 2025Baichuan-M2: Scaling Medical Capability with Large Verifier Systemcited by 55
  8. 2025GPTAQ: Efficient Finetuning-Free Quantization for Asymmetric Calibrationcited by 54
  9. 2025Optimizing Large Language Model Training Using FP4 Quantizationcited by 53
  10. 2024The Super Weight in Large Language Modelscited by 51
  11. 2025MiniCPM4: Ultra-Efficient LLMs on End Devicescited by 51
  12. 2024MARLIN: Mixed-Precision Auto-Regressive Parallel Inference on Large Language Modelscited by 48
  13. 2024MBQ: Modality-Balanced Quantization for Large Vision-Language Modelscited by 45
  14. 2025ParetoQ: Improving Scaling Laws in Extremely Low-bit LLM Quantizationcited by 44
  15. 2025RotateKV: Accurate and Robust 2-Bit KV Cache Quantization for LLMs via Outlier-Aware Adaptive Rotationscited by 42
  16. 2024SAM Decoding: Speculative Decoding via Suffix Automatoncited by 38
  17. 2025Quartet: Native FP4 Training Can Be Optimal for Large Language Modelscited by 37
  18. 2025Quantization Error Propagation: Revisiting Layer-Wise Post-Training Quantizationcited by 36
  19. 2025EXP-Bench: Can AI Conduct AI Research Experiments?cited by 36
  20. 2025MQuant: Unleashing the Inference Potential of Multimodal Large Language Models via Full Static Quantizationcited by 35
  21. 2024Basis Sharing: Cross-Layer Parameter Sharing for Large Language Model Compressioncited by 34
  22. 2024"Give Me BF16 or Give Me Death"? Accuracy-Performance Trade-Offs in LLM Quantizationcited by 34
  23. 2024Prefixing Attention Sinks can Mitigate Activation Outliers for Large Language Model Quantizationcited by 33
  24. 2024LLMC: Benchmarking Large Language Model Quantization with a Versatile Compression Toolkitcited by 32
  25. 2024ResQ: Mixed-Precision Quantization of Large Language Models with Low-Rank Residualscited by 32
  26. 2025MoEQuant: Enhancing Quantization for Mixture-of-Experts Large Language Models via Expert-Balanced Sampling and Affinity Guidancecited by 32
  27. 2024A Survey of Low-bit Large Language Models: Basics, Systems, and Algorithmscited by 30
  28. 2025Bridging the Gap Between Promise and Performance for Microscaling FP4 Quantizationcited by 30
  29. 2024SpikeLLM: Scaling up Spiking Neural Network to Large Language Models via Saliency-based Spikingcited by 29
  30. 2025Four Over Six: More Accurate NVFP4 Quantization with Adaptive Block Scalingcited by 29
  31. 2025GuidedQuant: Large Language Model Quantization via Exploiting End Loss Guidancecited by 28
  32. 2024Fast Matrix Multiplications for Lookup Table-Quantized LLMscited by 27
  33. 2025MambaQuant: Quantizing the Mamba Family with Variance Aligned Rotation Methodscited by 26
  34. 2025MxMoE: Mixed-precision Quantization for MoE with Accuracy and Performance Co-Designcited by 26
  35. 2024Quamba: A Post-Training Quantization Recipe for Selective State Space Modelscited by 25
  36. 2026QuantVLA: Scale-Calibrated Post-Training Quantization for Vision-Language-Action Modelscited by 25
  37. 2026LongLive-2.0: An NVFP4 Parallel Infrastructure for Long Video Generationcited by 25
  38. 2025Qronos: Correcting the Past by Shaping the Future... in Post-Training Quantizationcited by 24

50 most cited of 598 All 598 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023SmoothQuant+: Accurate and Efficient 4-bit Post-Training WeightQuantization for LLM
  2. 2024Taming Sensitive Weights : Noise Perturbation Fine-tuning for Robust LLM Quantization
  3. 2025Qrazor: Reliable and Effortless 4-bit LLM Quantization by Significant Data Razoring
  4. 2026XFP: Quality-Targeted Adaptive Codebook Quantization with Sparse Outlier Separation for LLM Inference
  5. 2026PrismQuant: Optimal Null-Space Rotations for Grouped Quantizers
  6. 2023Fast and Efficient 2-bit LLM Inference on GPU: 2/4/16-bit in a Weight Matrix with Asynchronous Dequantization
  7. 2026HyQuant: Hybrid-Precision Quantization for LLM Attention
  8. 2024Unlocking Data-free Low-bit Quantization with Matrix Decomposition for KV Cache Compression