mlsanityData to 7 Oct 2026

Reasoning Concentrates Errors, and Self-Consistency Never Notices

arXiv 2609.32035 ↗First posted 25 Sep 2026cs.AIcited by 0

On the map: LLM reasoning → Reasoning benchmarks and evaluation also fits: Test-time compute and efficient thinking

Abstract (arXiv)

Self-consistency assumes that independent samples disagree when a model is unsure, so agreement is evidence of correctness. Holding weights fixed and toggling only a reasoning mode, over five benchmarks and 74,944 samples, we show that reasoning concentrates a model's errors: the probability that two independently drawn wrong answers coincide rises in all ten dataset-scale comparisons (p = 0.00098), and in nine of nine after restricting both arms to the problems each gets wrong. Where the answer space is unbounded, reasoning cuts the distinct answers produced to 0.43-0.65 of the non-reasoning count; where it is bounded, both arms hold an identical option set and reasoning concentrates mass on it instead, which no positional prior can explain at fixed weights. The aggregate cost is smaller than the mechanism predicts, because reasoning also shrinks the set of problems where answer diversity can decide anything, in ten of ten cells and by 2.7x; normalized for available headroom, both arms convert a quarter of it in domain. Confidence weighting does not recover what is left. Across 280 method-dataset-model combinations on eight models and five benchmarks, not one beats plain majority voting after correction; weighted voting agrees with it on 98.5% of problem-method pairs and is right 56.3% of the time on the rest; and a signal's direction can invert within fixed weights, with answer log-probability predicting correctness when reasoning is off and error when it is on. A learned six-signal combination gains nothing out of domain. Confidence signals should be evaluated on decisions, not on discrimination.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 8 of its 22 references that are in the map. First posted on arXiv 25 Sep 2026.

Builds on

8 of 22 references.

References · 8

  1. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  2. 2025Gemma 3 Technical Reportcited by 1,686
  3. 2025Olmo 3cited by 95
  4. 2025Too Consistent to Detect: A Study of Self-Consistent Errors in LLMscited by 18
  5. 2026Reported Confidence in LLMs Tracks Commitment More Than Correctnesscited by 8
  6. 2025UNCERTAINTY-LINE: Length-Invariant Estimation of Uncertainty for Large Language Modelscited by 5
  7. 2026When Self-Consistency Backfires: Majority Vote Hurts the Majority of Hard Science Problems for Small LLMscited by 3
  8. 2026BODHI: Do LLMs Branch Out and Discover Heterogeneous Inferences?cited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Does Inference Scaling Improve Reasoning Faithfulness? A Multi-Model Analysis of Self-Consistency Tradeoffs
  2. 2025Self-Consistency Is Losing Its Edge: Diminishing Returns and Rising Costs in Modern LLMs
  3. 2026Quantifying Consistency in LLM Logical Reasoning via Structural Uncertainty
  4. 2025Catch Me If You Can: How Smaller Reasoning Models Pretend to Reason with Mathematical Fidelity
  5. 2026How Uncertainty Estimation Scales with Sampling in Reasoning Models
  6. 2026Amplified Does Not Mean Predictive: Reasoning Behaviors in Thinking Models
  7. 2025Revisiting Self-Consistency from Dynamic Distributional Alignment Perspective on Answer Aggregation
  8. 2026The Chain Holds, the Answer Folds: Trace-Answer Dissociation in Reasoning Models Under Adversarial Pressure