mlsanityData to 14 Oct 2026

DuQuant: Distributing Outliers via Dual Transformation Makes Stronger Quantized LLMs

arXiv 2406.01721 ↗First posted 3 Jun 2024cs.CLcited by 167

Not on the map. Why?

Cited by 22 in Efficient LLM inference and serving, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Quantization of large language models (LLMs) faces significant challenges, particularly due to the presence of outlier activations that impede efficient low-bit representation. Traditional approaches predominantly address Normal Outliers, which are activations across all tokens with relatively large magnitudes. However, these methods struggle with smoothing Massive Outliers that display significantly larger values, which leads to significant performance degradation in low-bit quantization. In this paper, we introduce DuQuant, a novel approach that utilizes rotation and permutation transformations to more effectively mitigate both massive and normal outliers. First, DuQuant starts by constructing the rotation matrix, using specific outlier dimensions as prior knowledge, to redistribute outliers to adjacent channels by block-wise rotation. Second, We further employ a zigzag permutation to balance the distribution of outliers across blocks, thereby reducing block-wise variance. A subsequent rotation further smooths the activation landscape, enhancing model performance. DuQuant simplifies the quantization process and excels in managing outliers, outperforming the state-of-the-art baselines across various sizes and types of LLMs on multiple tasks, even with 4-bit weight-activation quantization. Our code is available at https://github.com/Hsu1023/DuQuant.

Key facts. Cited by 167 papers in the map (data through 14 Oct 2026). Builds on 68 of its 89 references that are in the map. First posted on arXiv 3 Jun 2024.

Builds on

68 of 89 references.

References · 68

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  7. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  8. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  9. 2023Mistral 7Bcited by 3,164
  10. 2016Pointer Sentinel Mixture Modelscited by 3,096
  11. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  12. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,329
38 more
  1. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  2. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
  3. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,079
  4. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  5. 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,593
  6. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  7. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
  8. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,121
  9. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 906
  10. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 781
  11. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 637
  12. 2024QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMscited by 634
  13. 2020Up or Down? Adaptive Rounding for Post-Training Quantizationcited by 621
  14. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  15. 2024MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?cited by 586
  16. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 558
  17. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 526
  18. 2021BRECQ: Pushing the Limit of Post-Training Quantization by Block Reconstructioncited by 479
  19. 2023OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Modelscited by 424
  20. 2023QuIP: 2-Bit Quantization of Large Language Models With Guaranteescited by 414
  21. 2024QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebookscited by 378
  22. 2023SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compressioncited by 367
  23. 2023LLM-QAT: Data-Free Quantization Aware Training for Large Language Modelscited by 324
  24. 2023SqueezeLLM: Dense-and-Sparse Quantizationcited by 294
  25. 2024Massive Activations in Large Language Modelscited by 294
  26. 2021Post-Training Quantization for Vision Transformercited by 278
  27. 2023Atom: Low-bit Quantization for Efficient and Accurate LLM Servingcited by 272
  28. 2024SVD-LLM: Truncation-aware Singular Value Decomposition for Large Language Model Compressioncited by 243
  29. 2023Efficient Large Language Models: A Surveycited by 186
  30. 2022Outlier Suppression: Pushing the Limit of Low-bit Transformer Language Modelscited by 181
  31. 2020BinaryBERT: Pushing the Limit of BERT Quantizationcited by 178
  32. 2020TernaryBERT: Distillation-aware Ultra-low Bit BERTcited by 148
  33. 2023RPTQ: Reorder-based Post-training Quantization for Large Language Modelscited by 116
  34. 2024ZipCache: Accurate and Efficient KV Cache Quantization with Salient Token Identificationcited by 93
  35. 2024BitDistiller: Unleashing the Potential of Sub-4-Bit LLMs via Self-Distillationcited by 84
  36. 2023PB-LLM: Partially Binarized Large Language Modelscited by 83
  37. 2022Compression of Generative Pre-trained Language Models via Quantizationcited by 79
  38. 2024PTQ4DiT: Post-training Quantization for Diffusion Transformerscited by 79

50 most cited of 68 All 68 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 19; 2025: 73; 2026: 75.

167 papers.

Sort:

By area:

  1. 2024When Attention Sink Emerges in Language Models: An Empirical Viewcited by 208
  2. 2024A Survey on Large Language Model Acceleration based on KV Cache Managementcited by 131
  3. 2024Scaling Laws with Vocabulary: Larger Models Deserve Larger Vocabulariescited by 118
  4. 2024ATP-LLaVA: Adaptive Token Pruning for Large Vision Language Modelscited by 89
  5. 2024VoCo-LLaMA: Towards Vision Compression with Large Language Modelscited by 79
  6. 2025Training-free and Adaptive Sparse Attention for Efficient Long Video Generationcited by 69
  7. 2025GPTAQ: Efficient Finetuning-Free Quantization for Asymmetric Calibrationcited by 60
  8. 2024Active-Dormant Attention Heads: Mechanistically Demystifying Extreme-Token Phenomena in LLMscited by 56
  9. 2025TokLIP: Marry Visual Tokens to CLIP for Multimodal Comprehension and Generationcited by 51
  10. 2025RotateKV: Accurate and Robust 2-Bit KV Cache Quantization for LLMs via Outlier-Aware Adaptive Rotationscited by 46
  11. 2024ResQ: Mixed-Precision Quantization of Large Language Models with Low-Rank Residualscited by 36
  12. 2024STBLLM: Breaking the 1-Bit Barrier with Structured Binary LLMscited by 31
38 more
  1. 2026QuantVLA: Scale-Calibrated Post-Training Quantization for Vision-Language-Action Modelscited by 30
  2. 2026Flow caching for autoregressive video generationcited by 25
  3. 2024Towards Accurate Post-Training Quantization of Vision Transformers via Error Reductioncited by 23
  4. 2025NestQuant: Nested Lattice Quantization for Matrix Products and LLMscited by 23
  5. 2025Quantization Meets dLLMs: A Systematic Study of Post-training Quantization for Diffusion LLMscited by 23
  6. 2025Block Rotation is All You Need for MXFP4 Quantizationcited by 20
  7. 2024DopQ-ViT: Towards Distribution-Friendly and Outlier-Aware Post-Training Quantization for Vision Transformerscited by 19
  8. 2024AMXFP4: Taming Activation Outliers with Asymmetric Microscaling Floating-Point for 4-bit LLM Inferencecited by 19
  9. 2025TR-DQ: Time-Rotation Diffusion Quantizationcited by 19
  10. 2024Happy: A Debiased Learning Framework for Continual Generalized Category Discoverycited by 18
  11. 2025PAROAttention: Pattern-Aware ReOrdering for Efficient Sparse and Quantized Attention in Visual Generation Modelscited by 17
  12. 2025DartQuant: Efficient Rotational Distribution Calibration for LLM Quantizationcited by 17
  13. 2025ParoQuant: Pairwise Rotation Quantization for Efficient Reasoning LLM Inferencecited by 17
  14. 2024BitNet a4.8: 4-bit Activations for 1-bit LLMscited by 16
  15. 2025FPTQuant: Function-Preserving Transforms for LLM Quantizationcited by 16
  16. 2025QSVD: Efficient Low-rank Approximation for Unified Query-Key-Value Weight Compression in Low-Precision Vision-Language Modelscited by 16
  17. 2025Oaken: Fast and Efficient LLM Serving with Online-Offline Hybrid KV Cache Quantizationcited by 14
  18. 2024BoA: Attention-aware Post-training Quantization without Backpropagationcited by 12
  19. 2024PrefixQuant: Eliminating Outliers by Prefixed Tokens for Large Language Models Quantizationcited by 12
  20. 2025KurTail : Kurtosis-based LLM Quantizationcited by 12
  21. 2025LRQ-DiT: Log-Rotation Post-Training Quantization of Diffusion Transformers for Image and Video Generationcited by 12
  22. 2026DyQ-VLA: Temporal-Dynamic-Aware Quantization for Embodied Vision-Language-Action Modelscited by 12
  23. 2025Identifying Sensitive Weights via Post-quantization Integralcited by 11
  24. 2025Low-bit Model Quantization for Deep Neural Networks: A Surveycited by 11
  25. 2025MicroMix: Efficient Mixed-Precision Quantization with Microscaling Formats for Large Language Modelscited by 11
  26. 2026Pushing the Limits of Block Rotations in Post-Training Quantizationcited by 11
  27. 2026Attention Sinks Are Provably Necessary in Softmax Transformers: Evidence from Trigger-Conditional Taskscited by 11
  28. 2025PLANETALIGN: A Comprehensive Python Library for Benchmarking Network Alignmentcited by 10
  29. 2025EAQuant: Enhancing Post-Training Quantization for MoE Models via Expert-Aware Optimizationcited by 10
  30. 2025Hidden Dynamics of Massive Activations in Transformer Trainingcited by 10
  31. 2025ButterflyQuant: Ultra-low-bit LLM Quantization through Learnable Orthogonal Butterfly Transformscited by 10
  32. 2025Hierarchical LoRA MoE for Efficient CTR Model Scalingcited by 10
  33. 2025Harnessing Consistency for Robust Test-Time LLM Ensemblecited by 10
  34. 2025SpecQuant: Spectral Decomposition and Adaptive Truncation for Ultra-Low-Bit LLMs Quantizationcited by 10
  35. 2026Benchmarking Post-Training Quantization of Large Language Models under Microscaling Floating Point Formatscited by 10
  36. 2026Prune as You Generate: Online Rollout Pruning for Faster and Better RLVRcited by 10
  37. 2025ML-SpecQD: Multi-Level Speculative Decoding with Quantized Draftscited by 9
  38. 2024InterFormer: Effective Heterogeneous Interaction Learning for Click-Through Rate Predictioncited by 8

50 most cited of 167 All 167 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024OutlierTune: Efficient Channel-Wise Quantization for Large Language Models
  2. 2026MosaicQuant: Inlier-Outlier Disaggregation for Unified 4-Bit LLM Quantization
  3. 2025Rethinking the Outlier Distribution in Large Language Models: An In-depth Study
  4. 2025ICQuant: Index Coding enables Low-bit LLM Quantization
  5. 2023Rethinking Channel Dimensions to Isolate Outliers for Low-bit Weight Quantization of Large Language Models
  6. 2024DFRot: Achieving Outlier-Free and Massive Activation-Free for Rotated LLMs with Refined Rotation
  7. 2024EasyQuant: An Efficient Data-free Quantization Algorithm for LLMs
  8. 2025Outlier Smoothing with Closed-Form Rotations for W4A4 Large Language Model Quantization