mlsanityData to 14 Oct 2026

FlatQuant: Flatness Matters for LLM Quantization

arXiv 2410.09426 ↗First posted 12 Oct 2024cs.CL cs.LGcited by 131

Not on the map. Why?

Abstract (arXiv)

Recently, quantization has been widely used for the compression and acceleration of large language models (LLMs). Due to the outliers in LLMs, it is crucial to flatten weights and activations to minimize quantization error with equally spaced quantization points. Prior research explores various pre-quantization transformations to suppress outliers, such as per-channel scaling and Hadamard transformation. However, we observe that these transformed weights and activations can still exhibit steep and dispersed distributions. In this paper, we propose FlatQuant (Fast and Learnable Affine Transformation), a new post-training quantization approach that enhances the flatness of weights and activations. Our approach identifies optimal affine transformations for each linear layer, calibrated in hours via a lightweight objective. To reduce runtime overhead of affine transformation, we apply Kronecker product with two lightweight matrices, and fuse all operations in FlatQuant into a single kernel. Extensive experiments demonstrate that FlatQuant establishes a new state-of-the-art benchmark for quantization. For example, it achieves less than 1% accuracy drop for W4A4 quantization on the LLaMA-3-70B model, surpassing SpinQuant by 7.5%. Additionally, it provides up to 2.3x prefill speedup and 1.7x decoding speedup compared to the FP16 model. Code is available at: https://github.com/ruikangliu/FlatQuant.

Key facts. Cited by 131 papers in the map (data through 14 Oct 2026). Builds on 41 of its 54 references that are in the map. First posted on arXiv 12 Oct 2024.

Builds on

41 of 54 references.

References · 41

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  6. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  7. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  8. 2023Mistral 7Bcited by 3,164
  9. 2016Pointer Sentinel Mixture Modelscited by 3,096
  10. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  11. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,218
  12. 2024Qwen2 Technical Reportcited by 2,120
29 more
  1. 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,593
  2. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
  3. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 992
  4. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 800
  5. 2023LLM-Pruner: On the Structural Pruning of Large Language Modelscited by 650
  6. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 637
  7. 2024QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMscited by 634
  8. 2020Up or Down? Adaptive Rounding for Post-Training Quantizationcited by 621
  9. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 591
  10. 2021BRECQ: Pushing the Limit of Post-Training Quantization by Block Reconstructioncited by 479
  11. 2024SpinQuant: LLM quantization with learned rotationscited by 471
  12. 2023OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Modelscited by 424
  13. 2023QuIP: 2-Bit Quantization of Large Language Models With Guaranteescited by 414
  14. 2024QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebookscited by 378
  15. 2019Q-BERT: Hessian Based Ultra Low Precision Quantization of BERTcited by 359
  16. 2023SqueezeLLM: Dense-and-Sparse Quantizationcited by 294
  17. 2024Massive Activations in Large Language Modelscited by 294
  18. 2023Atom: Low-bit Quantization for Efficient and Accurate LLM Servingcited by 272
  19. 2024QServe: W4A8KV4 Quantization and System Co-design for Efficient LLM Servingcited by 208
  20. 2022Outlier Suppression: Pushing the Limit of Low-bit Transformer Language Modelscited by 181
  21. 2020BinaryBERT: Pushing the Limit of BERT Quantizationcited by 178
  22. 2024Evaluating Quantized Large Language Modelscited by 77
  23. 2024AffineQuant: Affine Transformation Quantization for Large Language Modelscited by 67
  24. 2020Robust Quantization: One Model to Rule Them Allcited by 63
  25. 2023Outlier Suppression+: Accurate quantization of large language models by equivalent and optimal shifting and scalingcited by 61
  26. 2023Training Transformers with 4-bit Integerscited by 60
  27. 2024IntactKV: Improving Large Language Model Quantization by Keeping Pivot Tokens Intactcited by 52
  28. 2024SKVQ: Sliding-window Key and Value Cache Quantization for Large Language Modelscited by 44
  29. 2021Towards Efficient Post-training Quantization of Pre-trained Language Modelscited by 39

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 30; 2026: 99.

131 papers.

Sort:

By area:

  1. 2024A Survey on Large Language Model Acceleration based on KV Cache Managementcited by 131
  2. 2025Quantization Hurts Reasoning? An Empirical Study on Quantized Reasoning Modelscited by 73
  3. 2024MBQ: Modality-Balanced Quantization for Large Vision-Language Modelscited by 45
  4. 2026QuantVLA: Scale-Calibrated Post-Training Quantization for Vision-Language-Action Modelscited by 30
  5. 2026Quantization-Aware Distillation for NVFP4 Inference Accuracy Recoverycited by 22
  6. 2025Block Rotation is All You Need for MXFP4 Quantizationcited by 20
  7. 2025ParoQuant: Pairwise Rotation Quantization for Efficient Reasoning LLM Inferencecited by 17
  8. 2025FPTQuant: Function-Preserving Transforms for LLM Quantizationcited by 16
  9. 2025WUSH: Near-Optimal Adaptive Transforms for LLM Quantizationcited by 15
  10. 2026ARCQuant: Boosting NVFP4 Quantization with Augmented Residual Channels for LLMscited by 15
  11. 2025S2Q-VDiT: Accurate Quantized Video Diffusion Transformer with Salient Data and Sparse Token Distillationcited by 13
  12. 2026Pushing the Limits of Block Rotations in Post-Training Quantizationcited by 11
38 more
  1. 2025ButterflyQuant: Ultra-low-bit LLM Quantization through Learnable Orthogonal Butterfly Transformscited by 10
  2. 2026What Makes Low-Bit Quantization-Aware Training Work for Reasoning LLMs? A Systematic Studycited by 10
  3. 2026Quantized Reasoning Models Think They Need to Think Longer, but They Do Notcited by 10
  4. 2025Preserving LLM Capabilities through Calibration Data Curation: From Analysis to Optimizationcited by 9
  5. 2025Unifying Block-wise PTQ and Distillation-based QAT for Progressive Quantization toward 2-bit Instruction-Tuned LLMscited by 8
  6. 2025Q-Palette: Fractional-Bit Quantizers Toward Optimal Bit Allocation for Efficient LLM Deploymentcited by 8
  7. 2026KBVQ-MoE: KLT-guided SVD with Bias-Corrected Vector Quantization for MoE Large Language Modelscited by 8
  8. 2026DuQuant++: Fine-grained Rotation Enhances Microscaling FP4 Quantizationcited by 8
  9. 2025QuantSparse: Comprehensively Compressing Video Diffusion Transformer with Model Quantization and Attention Sparsificationcited by 7
  10. 2026WaterSIC: Information-Theoretically (Near) Optimal Linear Layer Quantizationcited by 7
  11. 2026GEMQ: Global Expert-Level Mixed-Precision Quantization for MoE LLMscited by 7
  12. 2025Optimal Brain Restoration for Joint Quantization and Sparsification of LLMscited by 6
  13. 2025SINQ: Sinkhorn-Normalized Quantization for Calibration-Free Low-Precision LLM Weightscited by 6
  14. 2026BATQuant: Outlier-resilient MXFP4 Quantization via Learnable Block-wise Optimizationcited by 6
  15. 2026GSQ: Highly-Accurate Low-Precision Scalar Quantization for LLMs via Gumbel-Softmax Samplingcited by 6
  16. 2026LoopQ: Quantization for Looped Language Modelscited by 6
  17. 2026CAT-Q: Cost-efficient and Accurate Ternary Quantization for LLMscited by 6
  18. 2026OrbitQuant: Data-Agnostic Quantization for Image and Video Diffusion Transformerscited by 6
  19. 2025BASE-Q: Bias and Asymmetric Scaling Enhanced Rotational Quantization for Large Language Modelscited by 5
  20. 2025Prune&Comp: Free Lunch for Layer-Pruned LLMs via Iterative Pruning with Magnitude Compensationcited by 5
  21. 2026ECO: Quantized Training without Full-Precision Master Weightscited by 5
  22. 2026AdaTSQ: Pushing the Pareto Frontier of Diffusion Transformers via Temporal-Sensitivity Quantizationcited by 5
  23. 2026Dissecting Quantization Error: A Concentration-Alignment Perspectivecited by 5
  24. 2026MASQuant: Modality-Aware Smoothing Quantization for Multimodal Large Language Modelscited by 5
  25. 2026SliderQuant: Accurate Post-Training Quantization for LLMscited by 5
  26. 2025A Simple Linear Patch Revives Layer-Pruned Large Language Modelscited by 4
  27. 2025SmoothRot: Combining Channel-Wise Scaling and Rotation for Quantization-Friendly LLMscited by 4
  28. 2025BTC-LLM: Efficient Sub-1-Bit LLM Quantization via Learnable Transformation and Binary Codebookcited by 4
  29. 2026FreeAct: Freeing Activations for LLM Quantizationcited by 4
  30. 2026Adaptive Block-Scaled Data Typescited by 4
  31. 2025A Comprehensive Evaluation on Quantization Techniques for Large Language Modelscited by 3
  32. 2025Safety-Preserving PTQ via Contrastive Alignment Losscited by 3
  33. 2026An Empirical Study of World Model Quantizationcited by 3
  34. 2026TORQ: Two-Level Orthogonal Rotation for MXFP4 Quantizationcited by 3
  35. 2026TwinQuant: Learnable Subspace Decomposition for 4-Bit LLM Quantizationcited by 3
  36. 2026MosaicQuant: Inlier-Outlier Disaggregation for Unified 4-Bit LLM Quantizationcited by 3
  37. 2026KronQ: LLM Quantization via Kronecker-Factored Hessiancited by 3
  38. 2025Turning LLM Activations Quantization-Friendlycited by 2

50 most cited of 131 All 131 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024FlattenQuant: Breaking Through the Inference Compute-bound for Large Language Models with Per-tensor Quantization
  2. 2025OstQuant: Refining Large Language Model Quantization with Orthogonal and Scaling Transformations for Better Distribution Fitting
  3. 2024DuQuant: Distributing Outliers via Dual Transformation Makes Stronger Quantized LLMs
  4. 2024EasyQuant: An Efficient Data-free Quantization Algorithm for LLMs
  5. 2024PrefixQuant: Eliminating Outliers by Prefixed Tokens for Large Language Models Quantization
  6. 2026RUQuant: Towards Refining Uniform Quantization for Large Language Models
  7. 2024CDQuant: Greedy Coordinate Descent for Accurate LLM Quantization
  8. 2025ICQuant: Index Coding enables Low-bit LLM Quantization