mlsanityData to 14 Oct 2026

Quantized Reasoning Models Think They Need to Think Longer, but They Do Not

arXiv 2606.00206 ↗First posted 29 May 2026cs.LGcited by 10

On the map: LLM reasoning → Test-time compute and efficient thinking

Abstract (arXiv)

Post-training quantization (PTQ) is widely used to deploy large language models efficiently, but its effect on reasoning models is not well understood. Across math, coding, and science QA, we find that aggressive PTQ reduces accuracy while increasing chain-of-thought (CoT) length. Surprisingly, we show that in up to 52% of the quantized models' failures, models reach the right answer in intermediate reasoning steps but do not output it as a final answer. To understand why quantization leads to this increase in overthinking errors, we measure the token-level KL divergence between quantized and full-precision output distributions. Positions with high KL divergence correlate strongly with high next-token entropy, and at these positions quantized models disproportionately sample overthinking markers such as "wait", "but", and "alternatively". We show that simply introducing a training-free logit penalty on a curated set of overthinking markers can reduce CoT length by 12--23% while preserving or improving accuracy across 5 models (1.5B-32B parameters), 3 quantization methods, and 5 benchmarks, yielding a favorable Pareto frontier of accuracy against reasoning cost compared to penalizing other token sets. Overthinking errors produced by quantized models are particularly reduced by up to 58%.

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 34 of its 39 references that are in the map. First posted on arXiv 29 May 2026.

Builds on

34 of 39 references.

Closest ancestors · 10

Same direction, newest first

  1. 2025Wait, Wait, Wait... Why Do Reasoning Models Loop?15 Dec
  2. 2025Don't Overthink it. Preferring Shorter Thinking Chains for Improved LLM Reasoning23 May
  3. 2025Let LRMs Break Free from Overthinking via Self-Braking Tuning20 May
  4. 2025Between Underthinking and Overthinking: An Empirical Study of Reasoning Length and correctness in LLMs30 Apr
  5. 2025Dynamic Early Exit in Reasoning Models22 Apr
  6. 2025Missing Premise exacerbates Overthinking: Are Reasoning Models losing Critical Thinking Skill?9 Apr
  7. 2025Quantization Hurts Reasoning? An Empirical Study on Quantized Reasoning Models7 Apr
  8. 2025Stop Overthinking: A Survey on Efficient Reasoning for Large Language Models20 Mar
  9. 2025Quantization Meets Reasoning: Exploring LLM Low-Bit Quantization Degradation for Mathematical Reasoning6 Jan
  10. 2024Do NOT Think That Much for 2+3=? On the Overthinking of o1-Like LLMs30 Dec

Other references · 24

  1. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  2. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  3. 2016Pointer Sentinel Mixture Modelscited by 3,096
  4. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  5. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  6. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 2,004
  7. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  8. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
  9. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,298
  10. 2025OpenAI GPT-5 System Cardcited by 989
  11. 2024QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMscited by 634
  12. 2025Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoningcited by 559
12 more
  1. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 526
  2. 2024SpinQuant: LLM quantization with learned rotationscited by 471
  3. 2023QuIP: 2-Bit Quantization of Large Language Models With Guaranteescited by 414
  4. 2024QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebookscited by 378
  5. 2020Characterising Bias in Compressed Modelscited by 167
  6. 2026Beyond Benchmarks: MathArena as an Evaluation Platform for Mathematics with LLMscited by 137
  7. 2024FlatQuant: Flatness Matters for LLM Quantizationcited by 131
  8. 2021Differentiable Model Compression via Pseudo Quantization Noisecited by 45
  9. 2024Unlocking Tokens as Data Points for Generalization Bounds on Larger Language Modelscited by 25
  10. 2025Does quantization affect models' performance on long-context tasks?cited by 10
  11. 2026Investigating Social Bias Changes in Quantized Language Modelscited by 5
  12. 2026MCEL: Margin-Based Cross-Entropy Loss for Error-Tolerant Quantized Neural Networkscited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 10.

10 papers.

Similar, not cited

Close in wording, no citation either way

  1. 2026RATIO: Reasoning Analysis and Token-level Inference Optimization for Quantized Reasoning Models
  2. 2026Quantization Inflates Reasoning: Token Inflation as a Hidden Cost of Low-Bit Reasoning Models
  3. 2025Large Reasoning Models are not thinking straight: on the unreliability of thinking trajectories
  4. 2026Silent Failures in Quantized LLM Reasoning: A Taxonomy-Based Analysis of Hollow Convergence and Failure Mode Shifts
  5. 2026QuantLRM: Quantization of Large Reasoning Models via Fine-Tuning Signals
  6. 2025Quantization Meets Reasoning: Exploring and Mitigating Degradation of Low-Bit LLMs in Mathematical Reasoning
  7. 2026What Makes Low-Bit Quantization-Aware Training Work for Reasoning LLMs? A Systematic Study
  8. 2026Calibration Drift Under Reasoning: How Chain-of-Thought Budgets Induce Overconfidence in Large Language Models