# Quantization Meets Reasoning: Exploring LLM Low-Bit Quantization Degradation for Mathematical Reasoning

arXiv: https://arxiv.org/abs/2501.03035 · HTML: https://mlsanity.com/paper/2501.03035/

## Key facts

- Cited by 37 papers in the map (data through 14 Oct 2026).
- Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning.
- Builds on 20 of its 41 references that are in the map.
- First posted on arXiv 6 Jan 2025.

## Placed in

- LLM reasoning / [Test-time compute and efficient thinking](https://mlsanity.com/lr/test-time-compute-and-efficient-thinking/)

## Builds on (most cited first, 20 of 20)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [Let's Verify Step by Step](https://mlsanity.com/paper/2305.20050/) (2023)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [Training Compute-Optimal Large Language Models](https://mlsanity.com/paper/2203.15556/) (2022)
- [Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding](https://mlsanity.com/paper/1510.00149/) (2015)
- [GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers](https://mlsanity.com/paper/2210.17323/) (2022)
- [TinyBERT: Distilling BERT for Natural Language Understanding](https://mlsanity.com/paper/1909.10351/) (2019)
- [The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits](https://mlsanity.com/paper/2402.17764/) (2024)
- [Faithful Reasoning Using Large Language Models](https://mlsanity.com/paper/2208.14271/) (2022)
- [AgentSims: An Open-Source Sandbox for Large Language Model Evaluation](https://mlsanity.com/paper/2308.04026/) (2023)
- [Neural Networks with Few Multiplications](https://mlsanity.com/paper/1510.03009/) (2015)
- [DQ-LoRe: Dual Queries with Low Rank Approximation Re-ranking for In-Context Learning](https://mlsanity.com/paper/2310.02954/) (2023)
- [LLM-NEO: Parameter Efficient Knowledge Distillation for Large Language Models](https://mlsanity.com/paper/2411.06839/) (2024)

## Built on by (most cited first, 30 of 37)

- [Quantization Hurts Reasoning? An Empirical Study on Quantized Reasoning Models](https://mlsanity.com/paper/2504.04823/) (2025)
- [Low-Precision Training of Large Language Models: Methods, Challenges, and Opportunities](https://mlsanity.com/paper/2505.01043/) (2025)
- [Scaling LLM Test-Time Compute with Mobile NPU on Smartphones](https://mlsanity.com/paper/2509.23324/) (2025)
- [Does quantization affect models' performance on long-context tasks?](https://mlsanity.com/paper/2505.20276/) (2025)
- [What Makes Low-Bit Quantization-Aware Training Work for Reasoning LLMs? A Systematic Study](https://mlsanity.com/paper/2601.14888/) (2026)
- [Quantized Reasoning Models Think They Need to Think Longer, but They Do Not](https://mlsanity.com/paper/2606.00206/) (2026)
- [Is Finer Better? The Limits of Microscaling Formats in Large Language Models](https://mlsanity.com/paper/2601.19026/) (2026)
- [ZipServ: Fast and Memory-Efficient LLM Inference with Hardware-Aware Lossless Compression](https://mlsanity.com/paper/2603.17435/) (2026)
- [LeanK: Learnable K Cache Channel Pruning for Efficient Decoding](https://mlsanity.com/paper/2508.02215/) (2025)
- [AngelSlim: A more accessible, comprehensive, and efficient toolkit for large model compression](https://mlsanity.com/paper/2602.21233/) (2026)
- [Mixed-Precision Quantization for Language Models: Techniques and Prospects](https://mlsanity.com/paper/2510.16805/) (2025)
- [Mix-Quant: Quantized Prefilling, Precise Decoding for Agentic LLMs](https://mlsanity.com/paper/2605.20315/) (2026)
- [The Illusion of Equivalency: Statistical Characterization of Quantization Effects in LLMs](https://mlsanity.com/paper/2607.08734/) (2026)
- [Quantized Large Language Models in Biomedical Natural Language Processing: Evaluation and Recommendation](https://mlsanity.com/paper/2509.04534/) (2025)
- [Towards EnergyGPT: A Large Language Model Specialized for the Energy Sector](https://mlsanity.com/paper/2509.07177/) (2025)
- [ReQAT: Achieving Full-Precision Reasoning Accuracy with 4-bit Floating-Point Quantization-Aware Training](https://mlsanity.com/paper/2606.15682/) (2026)
- [Does Acceleration Cause Hidden Instability in Vision Language Models? Uncovering Instance-Level Divergence Through a Large-Scale Empirical Study](https://mlsanity.com/paper/2503.06794/) (2025)
- [Quantitative Analysis of Performance Drop in DeepSeek Model Quantization](https://mlsanity.com/paper/2505.02390/) (2025)
- [The Quantization Trap: Breaking Linear Scaling Laws in Multi-Hop Reasoning](https://mlsanity.com/paper/2602.13595/) (2026)
- [Reasoning-preserved Efficient Distillation of Large Language Models via Activation-aware Initialization](https://mlsanity.com/paper/2605.29327/) (2026)
- [Quantizing Recursive Reasoning Models](https://mlsanity.com/paper/2607.16237/) (2026)
- [From Quarter to All: Accelerating Speculative LLM Decoding via Floating-Point Exponent Remapping and Parameter Sharing](https://mlsanity.com/paper/2510.18525/) (2025)
- [Performance Trade-offs of Optimizing Small Language Models for E-Commerce](https://mlsanity.com/paper/2510.21970/) (2025)
- [Post-Training Quantization of OpenPangu Models for Efficient Deployment on Atlas A2](https://mlsanity.com/paper/2512.23367/) (2025)
- [A Case Study of Selected PTQ Baselines for Reasoning LLMs on Ascend NPU](https://mlsanity.com/paper/2602.17693/) (2026)
- [Rudder: Steering Prefetching in Distributed GNN Training using LLM Agents](https://mlsanity.com/paper/2602.23556/) (2026)
- [Different Prompts, Different Ranks: Prompt-aware Dynamic Rank Selection for SVD-based LLM Compression](https://mlsanity.com/paper/2605.08568/) (2026)
- [LAQuant: A Simple Overhead-free Large Reasoning Model Quantization by Layer-wise Lookahead Loss](https://mlsanity.com/paper/2605.08755/) (2026)
- [Beyond Activation Alignment:The Alignment-Diversity Tradeoff in Task-Aware LLM Quantization](https://mlsanity.com/paper/2607.00908/) (2026)
- [QUASAR: Lowering the Loss Floor of Quantization-Aware Training with Loss-Aware Reconstruction](https://mlsanity.com/paper/2608.13966/) (2026)

## Abstract (arXiv)

Large language models have achieved significant advancements in complex mathematical reasoning benchmarks, such as MATH. However, their substantial computational requirements present challenges for practical deployment. Model quantization has emerged as an effective strategy to reduce memory usage and computational costs by employing lower precision and bit-width representations. In this study, we systematically evaluate the impact of quantization on mathematical reasoning tasks. Our results demonstrate that aggressive quantization methods like AWQ and GPTQ introduce up to 32.39% accuracy degradation (average 11.31%) on Llama-3 models, particularly in numerical computation and reasoning planning. To address this, we introduce a multidimensional evaluation framework combining qualitative capability analysis and quantitative error assessment. We further develop targeted recovery strategies, showing that fine-tuning quantized models on only 545 task-specific examples for 3 minutes on 4 GPUs effectively restores reasoning capabilities to near full-precision levels. Additionally, our error assessment pipeline achieves 98.9% accuracy in diagnosing and localizing errors across 3,366 failure cases, providing actionable insights for mitigating quantization-induced degradation.