Coherence Under Commitment: Probing Generalization and Vacuous Memorization in LLM Logical Reasoning
On the map: LLM reasoning → Reasoning benchmarks and evaluation
Abstract (arXiv)
Large language models (LLMs) deployed for logical reasoning in knowledge-intensive domains exhibit a subtle but critical failure: coherence can be vacuously achieved through systematic abstention. A model that withholds commitment to either entailment or refutation satisfies negation consistency while providing no utility. We introduce Coherence Under Commitment (CUC), a dual-query evaluation paradigm that jointly measures consistency and decisiveness. CUC contributes three innovations: (1) a commitment score c(varphi) = p(varphi) + p(lnotvarphi) quantifying probability mass allocated to decisive outcomes; (2) a deterministic elicitation protocol via normalized YES/NO log probabilities, eliminating sampling variance; and (3) a 3-way decision framework (True/False/Uncertain) operationalizing the coherence-commitment trade-off into metrics. Experiments on four open-weight LLMs (1B-3B) across 204 FOLIO examples expose a sharp frontier. Qwen2.5-3B achieves near-zero contradiction (mathbbE[vneg]=0.025) but only 7.4% coverage, while TinyLlama-1.1B reaches 79.4% coverage with violations on every example. Coherence-only evaluation would rank the abstaining model first; CUC exposes this as vacuous, and the frontier generalizes to LogiQA~v2 (ρ=0.97). We argue that evaluation must report both coherence and non-vacuous commitment and release a toolkit for standardized assessment.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 15 of its 20 references that are in the map. First posted on arXiv 19 Jun 2026.
Builds on
15 of 20 references.
Closest ancestors · 1
Same direction, newest first
Other references · 14
- 2024The Llama 3 Herd of Modelscited by 14,715
- 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
- 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
- 2017On Calibration of Modern Neural Networkscited by 4,960
- 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
- 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
- 2022Survey of Hallucination in Natural Language Generationcited by 2,075
- 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,998
- 2022Language Models (Mostly) Know What They Knowcited by 1,856
- 2023Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Promptingcited by 1,122
- 2017Selective Classification for Deep Neural Networkscited by 852
- 2023SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Modelscited by 810
2 more
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026State commitment learning: training language models to distinguish computation from memory
- 2026Understanding Reasoning in LLMs through Strategic Information Allocation under Uncertainty
- 2026Can We Trust LLM's Logic? Quantifying Uncertainty, Coherence, and Robustness via a Graph-Based Framework
- 2026Towards a Mechanistic Understanding of Propositional Logical Reasoning in Large Language Models
- 2024Conditional and Modal Reasoning in Large Language Models
- 2026The Tell-Tale Trace: Detecting Reasoning Failures in LLMs Using Chain-of-Thought Dynamics
- 2025From Chaos to Order: The Atomic Reasoner Framework for Fine-grained Reasoning in Large Language Models
- 2026Beyond the Commitment Boundary: Probing Epiphenomenal Chain-of-Thought in Large Reasoning Models