mlsanityData to 7 Oct 2026

Quantization Meets Reasoning: Exploring LLM Low-Bit Quantization Degradation for Mathematical Reasoning

arXiv 2501.03035 ↗First posted 6 Jan 2025cs.CL cs.AIcited by 36

On the map: LLM reasoning → Test-time compute and efficient thinking

Abstract (arXiv)

Large language models have achieved significant advancements in complex mathematical reasoning benchmarks, such as MATH. However, their substantial computational requirements present challenges for practical deployment. Model quantization has emerged as an effective strategy to reduce memory usage and computational costs by employing lower precision and bit-width representations. In this study, we systematically evaluate the impact of quantization on mathematical reasoning tasks. Our results demonstrate that aggressive quantization methods like AWQ and GPTQ introduce up to 32.39% accuracy degradation (average 11.31%) on Llama-3 models, particularly in numerical computation and reasoning planning. To address this, we introduce a multidimensional evaluation framework combining qualitative capability analysis and quantitative error assessment. We further develop targeted recovery strategies, showing that fine-tuning quantized models on only 545 task-specific examples for 3 minutes on 4 GPUs effectively restores reasoning capabilities to near full-precision levels. Additionally, our error assessment pipeline achieves 98.9% accuracy in diagnosing and localizing errors across 3,366 failure cases, providing actionable insights for mitigating quantization-induced degradation.

Key facts. Cited by 36 papers in the map (data through 7 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 20 of its 41 references that are in the map. First posted on arXiv 6 Jan 2025.

Builds on

20 of 41 references.

References · 20

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  6. 2020Scaling Laws for Neural Language Modelscited by 6,411
  7. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  8. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  9. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  10. 2023Let's Verify Step by Stepcited by 3,598
  11. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  12. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,291
8 more
  1. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  2. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,153
  3. 2024The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bitscited by 289
  4. 2022Faithful Reasoning Using Large Language Modelscited by 130
  5. 2023AgentSims: An Open-Source Sandbox for Large Language Model Evaluationcited by 86
  6. 2015Neural Networks with Few Multiplicationscited by 46
  7. 2023DQ-LoRe: Dual Queries with Low Rank Approximation Re-ranking for In-Context Learningcited by 5
  8. 2024LLM-NEO: Parameter Efficient Knowledge Distillation for Large Language Modelscited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 15; 2026: 21.

36 papers.

Sort:

By area:

  1. 2025Quantization Hurts Reasoning? An Empirical Study on Quantized Reasoning Modelscited by 68
  2. 2025Low-Precision Training of Large Language Models: Methods, Challenges, and Opportunitiescited by 11
  3. 2025Does quantization affect models' performance on long-context tasks?cited by 10
  4. 2026Quantized Reasoning Models Think They Need to Think Longer, but They Do Notcited by 10
  5. 2025Scaling LLM Test-Time Compute with Mobile NPU on Smartphonescited by 9
  6. 2026What Makes Low-Bit Quantization-Aware Training Work for Reasoning LLMs? A Systematic Studycited by 9
  7. 2026ZipServ: Fast and Memory-Efficient LLM Inference with Hardware-Aware Lossless Compressioncited by 8
  8. 2026Is Finer Better? The Limits of Microscaling Formats in Large Language Modelscited by 7
  9. 2025LeanK: Learnable K Cache Channel Pruning for Efficient Decodingcited by 5
  10. 2026AngelSlim: A more accessible, comprehensive, and efficient toolkit for large model compressioncited by 5
  11. 2025Mixed-Precision Quantization for Language Models: Techniques and Prospectscited by 4
  12. 2025Quantized Large Language Models in Biomedical Natural Language Processing: Evaluation and Recommendationcited by 3
24 more
  1. 2025Towards EnergyGPT: A Large Language Model Specialized for the Energy Sectorcited by 3
  2. 2026Mix-Quant: Quantized Prefilling, Precise Decoding for Agentic LLMscited by 3
  3. 2026The Illusion of Equivalency: Statistical Characterization of Quantization Effects in LLMscited by 3
  4. 2025Does Acceleration Cause Hidden Instability in Vision Language Models? Uncovering Instance-Level Divergence Through a Large-Scale Empirical Studycited by 2
  5. 2025Quantitative Analysis of Performance Drop in DeepSeek Model Quantizationcited by 2
  6. 2026The Quantization Trap: Breaking Linear Scaling Laws in Multi-Hop Reasoningcited by 2
  7. 2026Reasoning-preserved Efficient Distillation of Large Language Models via Activation-aware Initializationcited by 2
  8. 2026ReQAT: Achieving Full-Precision Reasoning Accuracy with 4-bit Floating-Point Quantization-Aware Trainingcited by 2
  9. 2026Quantizing Recursive Reasoning Modelscited by 2
  10. 2025From Quarter to All: Accelerating Speculative LLM Decoding via Floating-Point Exponent Remapping and Parameter Sharingcited by 1
  11. 2025Performance Trade-offs of Optimizing Small Language Models for E-Commercecited by 1
  12. 2025Post-Training Quantization of OpenPangu Models for Efficient Deployment on Atlas A2cited by 1
  13. 2026A Case Study of Selected PTQ Baselines for Reasoning LLMs on Ascend NPUcited by 1
  14. 2026LAQuant: A Simple Overhead-free Large Reasoning Model Quantization by Layer-wise Lookahead Losscited by 1
  15. 2026Beyond Activation Alignment:The Alignment-Diversity Tradeoff in Task-Aware LLM Quantizationcited by 1
  16. 2026QUASAR: Lowering the Loss Floor of Quantization-Aware Training with Loss-Aware Reconstructioncited by 1
  17. 2025Compressed Causal Reasoning: Quantization and GraphRAG Effects on Interventional and Counterfactual Accuracycited by 0
  18. 2025Hierarchical Sparse Plus Low Rank Compression of LLMcited by 0
  19. 2026Rudder: Steering Prefetching in Distributed GNN Training using LLM Agentscited by 0
  20. 2026Different Prompts, Different Ranks: Prompt-aware Dynamic Rank Selection for SVD-based LLM Compressioncited by 0
  21. 2026BitNet Text Embeddingscited by 0
  22. 2026Quantization Degradation in Large Language Models: A Signal-Noise Perspectivecited by 0
  23. 2026Are You Sure You're Sure? Two Confounds in a Sycophancy Benchmarkcited by 0
  24. 2026RATIO: Reasoning Analysis and Token-level Inference Optimization for Quantized Reasoning Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Quantization Meets Reasoning: Exploring and Mitigating Degradation of Low-Bit LLMs in Mathematical Reasoning
  2. 2026Quantization Inflates Reasoning: Token Inflation as a Hidden Cost of Low-Bit Reasoning Models
  3. 2026Silent Failures in Quantized LLM Reasoning: A Taxonomy-Based Analysis of Hollow Convergence and Failure Mode Shifts
  4. 2025Mathematical Reasoning in Large Language Models: Assessing Logical and Arithmetic Errors across Wide Numerical Ranges
  5. 2026Attend to Your Own Thoughts: Breaking the Barrier for Post-Training Quantization of Reasoning LLMs through the Lens of 1.58-Bit Quantization
  6. 2026Mathematical Reasoning in Large Language Models: Benchmarks, Architectures, Evaluation, and Open Challenges
  7. 2026Extreme Low-Bit Inference in Reasoning Models: Failure Modes and Targeted Recovery
  8. 2025Understanding and Mitigating Numerical Sources of Nondeterminism in LLM Inference