Quantifying Consistency in LLM Logical Reasoning via Structural Uncertainty
On the map: LLM reasoning → Test-time compute and efficient thinking
Abstract (arXiv)
Large language models can arrive at the same answer through reasoning paths that are unstable, contradictory, or difficult to rank consistently -- a failure mode especially prevalent in multi-step deductive reasoning. Existing methods assess reliability primarily through output dispersion -- measuring how much sampled answers differ -- but this discards a complementary signal: whether the model can consistently rank competing reasoning candidates. We propose structural uncertainty, a consistency-aware framework derived from the stability of self-preference-induced rankings over sampled reasoning solutions. Given a query, we generate multiple candidate solutions and ask the model to judge pairwise preferences among its own outputs. We aggregate self-preferences into ranking distributions via Bradley-Terry modeling with PageRank, and decompose the signal into two entropy-based components: across-trial ranking instability and within-trial candidate ambiguity. Across five LLMs and eight benchmarks, structural signals provide information complementary to answer dispersion: on logical and mathematical reasoning tasks, the combination improves identification of unreliable instances, while on factual retrieval the structural signal collapses toward uniformity, diagnosing a regime boundary where reasoning-level consistency evaluation is uninformative. The two components relate differently to accuracy: within-trial ambiguity correlates positively with correctness -- consistent with settings where multiple plausible solution paths remain competitive -- while across-trial instability correlates negatively, signaling unreliable reasoning. Structural uncertainty is best understood not as a universal confidence estimator, but as a regime-sensitive evaluator of logical reasoning consistency.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 30 of its 45 references that are in the map. First posted on arXiv 15 Jun 2026.
Builds on
30 of 45 references.
Closest ancestors · 2
Same direction, newest first
- 2025CoT-UQ: Improving Response-wise Uncertainty Quantification in LLMs with Chain-of-Thought24 Feb
- 2022Language Models (Mostly) Know What They Know11 Jul
Other references · 28
- 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
- 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
- 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
- 2023Let's Verify Step by Stepcited by 3,598
- 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
- 2017What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?cited by 2,335
- 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,758
- 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 946
- 2023Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generationcited by 795
- 2023Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedbackcited by 788
- 2023Generating with Confidence: Uncertainty Quantification for Black-box Large Language Modelscited by 286
- 2024Semantic Entropy Probes: Robust and Cheap Hallucination Detection in LLMscited by 246
16 more
- 2024Kernel Language Entropy: Fine-grained Uncertainty Quantification for LLMs from Semantic Similaritiescited by 156
- 2024Benchmarking LLMs via Uncertainty Quantificationcited by 124
- 2024To Believe or Not to Believe Your LLMcited by 120
- 2024Relying on the Unreliable: The Impact of Language Models' Reluctance to Express Uncertaintycited by 106
- 2024Benchmarking Uncertainty Quantification Methods for Large Language Models with LM-Polygraphcited by 105
- 2024Calibrating Large Language Models with Sample Consistencycited by 72
- 2024Uncertainty in Language Models: Assessment through Rank-Calibrationcited by 40
- 2023Llamas Know What GPTs Don't Show: Surrogate Models for Confidence Estimationcited by 38
- 2025Estimating LLM Uncertainty with Evidencecited by 26
- 2024MAQA: Evaluating Uncertainty Quantification in LLMs Regarding Data Uncertaintycited by 25
- 2025Uncertainty Quantification of Large Language Models through Multi-Dimensional Responsescited by 17
- 2021Active Learning in Bayesian Neural Networks with Balanced Entropy Learning Principlecited by 14
- 2024Self-Consistency Boosts Calibration for Math Reasoningcited by 12
- 2025Language Models Prefer What They Know: Relative Confidence Estimation via Confidence Preferencescited by 12
- 2024Improving Instruction Following in Language Models through Proxy-Based Uncertainty Estimationcited by 6
- 2022Analytic Mutual Information in Bayesian Neural Networkscited by 5
Built on by
Citing papers per year: 2026: 2.
2 papers.
Similar, not cited
Close in wording, no citation either way
- 2026Can We Trust LLM's Logic? Quantifying Uncertainty, Coherence, and Robustness via a Graph-Based Framework
- 2026Does Inference Scaling Improve Reasoning Faithfulness? A Multi-Model Analysis of Self-Consistency Tradeoffs
- 2026Operadic consistency: a label-free signal for compositional reasoning failures in LLMs
- 2025Confidence Improves Self-Consistency in LLMs
- 2024Aligning with Logic: Measuring, Evaluating and Improving Logical Preference Consistency in Large Language Models
- 2025ReasonBENCH: Benchmarking the (In)Stability of LLM Reasoning
- 2025A Theoretical Study on Bridging Internal Probability and Self-Consistency for LLM Reasoning
- 2025Revisiting Self-Consistency from Dynamic Distributional Alignment Perspective on Answer Aggregation