mlsanityData to 14 Oct 2026

Towards Superior Quantization Accuracy: A Layer-sensitive Approach

arXiv 2503.06518 ↗First posted 9 Mar 2025cs.LG cs.AIcited by 8

On the map: LLM reasoning → Test-time compute and efficient thinking

Abstract (arXiv)

Large Vision and Language Models have exhibited remarkable human-like intelligence in tasks such as natural language comprehension, problem-solving, logical reasoning, and knowledge retrieval. However, training and serving these models require substantial computational resources, posing a significant barrier to their widespread application and further research. To mitigate this challenge, various model compression techniques have been developed to reduce computational requirements. Nevertheless, existing methods often employ uniform quantization configurations, failing to account for the varying difficulties across different layers in quantizing large neural network models. This paper tackles this issue by leveraging layer-sensitivity features, such as activation sensitivity and weight distribution Kurtosis, to identify layers that are challenging to quantize accurately and allocate additional memory budget. The proposed methods, named SensiBoost and KurtBoost, respectively, demonstrate notable improvement in quantization accuracy, achieving up to 9% lower perplexity with only a 2% increase in memory budget on LLama models compared to the baseline.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 19 of its 31 references that are in the map. First posted on arXiv 9 Mar 2025.

Builds on

19 of 31 references.

References · 19

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,611
  3. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  4. 2021Convex optimizationcited by 2,515
  5. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  6. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 591
  7. 2019Importance Estimation for Neural Network Pruningcited by 541
  8. 2021A White Paper on Neural Network Quantizationcited by 472
  9. 20218-bit Optimizers via Block-wise Quantizationcited by 403
  10. 2019Data-Free Quantization Through Weight Equalization and Bias Correctioncited by 336
  11. 2022Optimal Brain Compression: A Framework for Accurate Post-Training Quantization and Pruningcited by 322
  12. 2023SqueezeLLM: Dense-and-Sparse Quantizationcited by 294
7 more
  1. 2022Outlier Suppression: Pushing the Limit of Low-bit Transformer Language Modelscited by 181
  2. 2021BERT Busters: Outlier Dimensions that Disrupt Transformerscited by 112
  3. 2023LQ-LoRA: Low-rank Plus Quantized Matrix Decomposition for Efficient Language Model Finetuningcited by 92
  4. 2015Parallelizing the dual revised simplex methodcited by 89
  5. 2023Initializing Models with Larger Onescited by 39
  6. 2025Systematic Outliers in Large Language Modelscited by 33
  7. 2023Weight subcloning: direct initialization of transformers using larger pretrained onescited by 30

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 6.

8 papers.

  1. 2026Locate, Steer, and Improve: A Practical Survey of Actionable Mechanistic Interpretability in Large Language Modelscited by 27
  2. 2026Beyond Outliers: A Data-Free Layer-wise Mixed-Precision Quantization Approach Driven by Numerical and Structural Dual-Sensitivitycited by 5
  3. 2026From Signal Degradation to Computation Collapse: Uncovering the Two Failure Modes of LLM Quantizationcited by 5
  4. 2025You Had One Job: Per-Task Quantization Using LLMs' Hidden Representationscited by 3
  5. 2025SpikeFit: Towards Optimal Deployment of Spiking Networks on Neuromorphic Hardwarecited by 2
  6. 2026RaBiT: Residual-Aware Binarization Training for Accurate and Efficient LLMscited by 0
  7. 2026Minimizing the Hidden Cost of Scales: Graph-Guided Ultra-Low-Bit Quantization for Large Language Modelscited by 0
  8. 2026Component Type, Not Reconstruction Error, Predicts Attention Quantization Sensitivitycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Exploring Layer-wise Information Effectiveness for Post-Training Quantization in Small Language Models
  2. 2024Layer-Wise Quantization: A Pragmatic and Effective Method for Quantizing LLMs Beyond Integer Bit-Levels
  3. 2023QuantEase: Optimization-based Quantization for Language Models
  4. 2026Cross-Layer Error Compensation and Finite-Sample Feature-Statistics Matching for Extreme Low-Bit Quantization of Large Language Models
  5. 2024Exploring the Trade-Offs: Quantization Methods, Task Difficulty, and Model Size in Large Language Models From Edge to Giant
  6. 2024Magic for the Age of Quantized DNNs
  7. 2025Sensitivity-Aware Post-Training Quantization for Deep Neural Networks
  8. 2024LSAQ: Layer-Specific Adaptive Quantization for Large Language Model Deployment