Quantifying Cross-Query Contradictions in Multi-Query LLM Reasoning
On the map: LLM reasoning → Reasoning benchmarks and evaluation
Abstract (arXiv)
Large language models frequently produce mutually inconsistent answers when reasoning over multiple related queries. We study case-file logical consistency: maintaining a globally satisfiable belief state across interdependent queries. We introduce a benchmark of 390 multi-query reasoning instances with entailment/contradiction/unknown labels and propose set-level metrics including Case Satisfiability Rate, Contradiction Density and Revision Cost. Our solver-augmented approach extracts commitments, verifies global satisfiability and performs counterexample-guided repair. Across four reasoning domains, our method substantially reduces cross-query contradictions (SetCons: 0.56 to 0.94) while preserving per-query accuracy, demonstrating that global coherence is critical for robust multi-query reasoning.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 8 of its 13 references that are in the map. First posted on arXiv 16 Apr 2026.
Builds on
8 of 13 references.
Closest ancestors · 4
Same direction, newest first
- 2025MTR-Bench: A Comprehensive Benchmark for Multi-Turn Reasoning Evaluation21 May
- 2025SATBench: Benchmarking LLMs' Logical Reasoning via Automated Puzzle Generation from SAT Formulas20 May
- 2025JustLogic: A Comprehensive Benchmark for Evaluating Deductive Reasoning in Large Language Models24 Jan
- 2023Logic-LM: Empowering Large Language Models with Symbolic Solvers for Faithful Logical Reasoning20 May
Other references · 4
- 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
- 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
- 2024MT-Eval: A Multi-Turn Capabilities Evaluation Benchmark for Large Language Modelscited by 81
- 2024Logic-of-Thought: Injecting Logic into Contexts for Full Reasoning in Large Language Modelscited by 23
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2024Logical Consistency of Large Language Models in Fact-checking
- 2026Quantifying Consistency in LLM Logical Reasoning via Structural Uncertainty
- 2026Evaluating Robustness of Reasoning Models on Parameterized Logical Problems
- 2026Beyond Answer Voting: Fine-Grained Correction of LLM Reasoning through Graph Alignment across Sampled Traces
- 2026Satisfiability Solving with LLMs: A Matched-Pair Evaluation of Reasoning Capability
- 2026Coherence Under Commitment: Probing Generalization and Vacuous Memorization in LLM Logical Reasoning
- 2026Exploring Knowledge Conflicts for Faithful LLM Reasoning: Benchmark and Method
- 2025Beyond Correctness: Exposing LLM-generated Logical Flaws in Reasoning via Multi-step Automated Theorem Proving