# Quantifying Consistency in LLM Logical Reasoning via Structural Uncertainty

arXiv: https://arxiv.org/abs/2606.17312 · HTML: https://mlsanity.com/paper/2606.17312/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning.
- Builds on 30 of its 45 references that are in the map.
- First posted on arXiv 15 Jun 2026.

## Placed in

- LLM reasoning / [Test-time compute and efficient thinking](https://mlsanity.com/lr/test-time-compute-and-efficient-thinking/)

## Builds on (most cited first, 30 of 30)

- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [Let's Verify Step by Step](https://mlsanity.com/paper/2305.20050/) (2023)
- [HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering](https://mlsanity.com/paper/1809.09600/) (2018)
- [TruthfulQA: Measuring How Models Mimic Human Falsehoods](https://mlsanity.com/paper/2109.07958/) (2021)
- [What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?](https://mlsanity.com/paper/1703.04977/) (2017)
- [Language Models (Mostly) Know What They Know](https://mlsanity.com/paper/2207.05221/) (2022)
- [MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark](https://mlsanity.com/paper/2406.01574/) (2024)
- [Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs](https://mlsanity.com/paper/2306.13063/) (2023)
- [Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generation](https://mlsanity.com/paper/2302.09664/) (2023)
- [Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback](https://mlsanity.com/paper/2305.14975/) (2023)
- [Generating with Confidence: Uncertainty Quantification for Black-box Large Language Models](https://mlsanity.com/paper/2305.19187/) (2023)
- [Semantic Entropy Probes: Robust and Cheap Hallucination Detection in LLMs](https://mlsanity.com/paper/2406.15927/) (2024)
- [Kernel Language Entropy: Fine-grained Uncertainty Quantification for LLMs from Semantic Similarities](https://mlsanity.com/paper/2405.20003/) (2024)
- [Benchmarking LLMs via Uncertainty Quantification](https://mlsanity.com/paper/2401.12794/) (2024)
- [To Believe or Not to Believe Your LLM](https://mlsanity.com/paper/2406.02543/) (2024)
- [Relying on the Unreliable: The Impact of Language Models' Reluctance to Express Uncertainty](https://mlsanity.com/paper/2401.06730/) (2024)
- [Benchmarking Uncertainty Quantification Methods for Large Language Models with LM-Polygraph](https://mlsanity.com/paper/2406.15627/) (2024)
- [Calibrating Large Language Models with Sample Consistency](https://mlsanity.com/paper/2402.13904/) (2024)
- [Uncertainty in Language Models: Assessment through Rank-Calibration](https://mlsanity.com/paper/2404.03163/) (2024)
- [Llamas Know What GPTs Don't Show: Surrogate Models for Confidence Estimation](https://mlsanity.com/paper/2311.08877/) (2023)
- [MAQA: Evaluating Uncertainty Quantification in LLMs Regarding Data Uncertainty](https://mlsanity.com/paper/2408.06816/) (2024)
- [Estimating LLM Uncertainty with Evidence](https://mlsanity.com/paper/2502.00290/) (2025)
- [CoT-UQ: Improving Response-wise Uncertainty Quantification in LLMs with Chain-of-Thought](https://mlsanity.com/paper/2502.17214/) (2025)
- [Uncertainty Quantification of Large Language Models through Multi-Dimensional Responses](https://mlsanity.com/paper/2502.16820/) (2025)
- [Active Learning in Bayesian Neural Networks with Balanced Entropy Learning Principle](https://mlsanity.com/paper/2105.14559/) (2021)
- [Self-Consistency Boosts Calibration for Math Reasoning](https://mlsanity.com/paper/2403.09849/) (2024)
- [Language Models Prefer What They Know: Relative Confidence Estimation via Confidence Preferences](https://mlsanity.com/paper/2502.01126/) (2025)
- [Improving Instruction Following in Language Models through Proxy-Based Uncertainty Estimation](https://mlsanity.com/paper/2405.06424/) (2024)
- [Analytic Mutual Information in Bayesian Neural Networks](https://mlsanity.com/paper/2201.09815/) (2022)

## Built on by (most cited first, 2 of 2)

- [Every Wrong Answer Counts: Option-Level Psychometrics for LLM Multiple-Choice Benchmarks](https://mlsanity.com/paper/2608.02966/) (2026)
- [The Complexity Ceiling Benchmark: A Multi-Domain Evaluation of Sequential Reasoning Under Depth Scaling](https://mlsanity.com/paper/2606.29278/) (2026)

## Abstract (arXiv)

Large language models can arrive at the same answer through reasoning paths that are unstable, contradictory, or difficult to rank consistently -- a failure mode especially prevalent in multi-step deductive reasoning. Existing methods assess reliability primarily through output dispersion -- measuring how much sampled answers differ -- but this discards a complementary signal: whether the model can consistently rank competing reasoning candidates. We propose structural uncertainty, a consistency-aware framework derived from the stability of self-preference-induced rankings over sampled reasoning solutions. Given a query, we generate multiple candidate solutions and ask the model to judge pairwise preferences among its own outputs. We aggregate self-preferences into ranking distributions via Bradley-Terry modeling with PageRank, and decompose the signal into two entropy-based components: across-trial ranking instability and within-trial candidate ambiguity. Across five LLMs and eight benchmarks, structural signals provide information complementary to answer dispersion: on logical and mathematical reasoning tasks, the combination improves identification of unreliable instances, while on factual retrieval the structural signal collapses toward uniformity, diagnosing a regime boundary where reasoning-level consistency evaluation is uninformative. The two components relate differently to accuracy: within-trial ambiguity correlates positively with correctness -- consistent with settings where multiple plausible solution paths remain competitive -- while across-trial instability correlates negatively, signaling unreliable reasoning. Structural uncertainty is best understood not as a universal confidence estimator, but as a regime-sensitive evaluator of logical reasoning consistency.