mlsanityData to 14 Oct 2026

MergeQuant: Accurate 4-bit Static Quantization of Large Language Models by Channel-wise Calibration

arXiv 2503.07654 ↗First posted 7 Mar 2025cs.LGcited by 4

Not on the map. Why?

Abstract (arXiv)

Quantization has been widely used to compress and accelerate inference of large language models (LLMs). Existing methods focus on exploring the per-token dynamic calibration to ensure both inference acceleration and model accuracy under 4-bit quantization. However, in autoregressive generation inference of long sequences, the overhead of repeated dynamic quantization and dequantization steps becomes considerably expensive. In this work, we propose MergeQuant, an accurate and efficient per-channel static quantization framework. MergeQuant integrates the per-channel quantization steps with the corresponding scalings and linear mappings through a Quantization Step Migration (QSM) method, thereby eliminating the quantization overheads before and after matrix multiplication. Furthermore, in view of the significant differences between the different channel ranges, we propose dimensional reconstruction and adaptive clipping to address the non-uniformity of quantization scale factors and redistribute the channel variations to the subsequent modules to balance the parameter distribution under QSM. Within the static quantization setting of W4A4, MergeQuant reduces the accuracy gap on zero-shot tasks compared to FP16 baseline to 1.3 points on Llama-2-70B model. On Llama-2-7B model, MergeQuant achieves up to 1.77x speedup in decoding, and up to 2.06x speedup in end-to-end compared to FP16 baseline.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 47 of its 54 references that are in the map. First posted on arXiv 7 Mar 2025.

Builds on

47 of 54 references.

References · 47

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  6. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  7. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  8. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  9. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  10. 2016Pointer Sentinel Mixture Modelscited by 3,096
  11. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  12. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,079
35 more
  1. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  2. 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,593
  3. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
  4. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 906
  5. 2021A Survey of Quantization Methods for Efficient Neural Network Inferencecited by 815
  6. 2024QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMscited by 634
  7. 2020Up or Down? Adaptive Rounding for Post-Training Quantizationcited by 621
  8. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 575
  9. 2022Efficiently Scaling Transformer Inferencecited by 539
  10. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 526
  11. 2021BRECQ: Pushing the Limit of Post-Training Quantization by Block Reconstructioncited by 479
  12. 2024SpinQuant: LLM quantization with learned rotationscited by 471
  13. 2018PACT: Parameterized Clipping Activation for Quantized Neural Networkscited by 467
  14. 2018HAQ: Hardware-Aware Automated Quantization with Mixed Precisioncited by 443
  15. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Timecited by 439
  16. 2023OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Modelscited by 424
  17. 2023QuIP: 2-Bit Quantization of Large Language Models With Guaranteescited by 414
  18. 2019Q-BERT: Hessian Based Ultra Low Precision Quantization of BERTcited by 359
  19. 2022Data Distributional Properties Drive Emergent In-Context Learning in Transformerscited by 309
  20. 2018Model compression via distillation and quantizationcited by 289
  21. 2019Low-bit Quantization of Neural Networks for Efficient Inferencecited by 193
  22. 2022Outlier Suppression: Pushing the Limit of Low-bit Transformer Language Modelscited by 181
  23. 2020BinaryBERT: Pushing the Limit of BERT Quantizationcited by 178
  24. 2020TernaryBERT: Distillation-aware Ultra-low Bit BERTcited by 148
  25. 2023RPTQ: Reorder-based Post-training Quantization for Large Language Modelscited by 116
  26. 2021BERT Busters: Outlier Dimensions that Disrupt Transformerscited by 112
  27. 2023OliVe: Accelerating Large Language Models via Hardware-friendly Outlier-Victim Pair Quantizationcited by 92
  28. 2022Compression of Generative Pre-trained Language Models via Quantizationcited by 79
  29. 2023QLLM: Accurate and Efficient Low-Bitwidth Quantization for Large Language Modelscited by 64
  30. 2023Outlier Suppression+: Accurate quantization of large language models by equivalent and optimal shifting and scalingcited by 61
  31. 2020Towards Fully 8-bit Integer Inference for the Transformer Modelcited by 35
  32. 2020Positional Artefacts Propagate Through Masked Language Model Embeddingscited by 35
  33. 2022Fine-tuning Language Models over Slow Networks using Activation Compression with Guaranteescited by 30
  34. 2023Understanding INT4 Quantization for Transformer Models: Latency Speedup, Composability, and Failure Casescited by 23
  35. 2024OutlierTune: Efficient Channel-Wise Quantization for Large Language Modelscited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 1.

4 papers.

  1. 2025Mixed-Precision Quantization for Language Models: Techniques and Prospectscited by 5
  2. 2025ProfilingAgent: Profiling-Guided Agentic Reasoning for Adaptive Model Optimizationcited by 3
  3. 2025Outlier Smoothing with Closed-Form Rotations for W4A4 Large Language Model Quantizationcited by 1
  4. 2026Transforms for LLM Quantization: The Great Inversion and Format Co-Designcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025FlexQuant: A Flexible and Efficient Dynamic Precision Switching Framework for LLM Quantization
  2. 2025A Comprehensive Evaluation on Quantization Techniques for Large Language Models
  3. 2024QQQ: Quality Quattuor-Bit Quantization for Large Language Models
  4. 2025OstQuant: Refining Large Language Model Quantization with Orthogonal and Scaling Transformations for Better Distribution Fitting
  5. 2024FlattenQuant: Breaking Through the Inference Compute-bound for Large Language Models with Per-tensor Quantization
  6. 2024LeanQuant: Accurate and Scalable Large Language Model Quantization with Loss-error-aware Grid
  7. 2025PTQTP: Post-Training Quantization to Trit-Planes for Large Language Models
  8. 2026SFMP: Fine-Grained, Hardware-Friendly and Search-Free Mixed-Precision Quantization for Large Language Models