Compositional Consistency-Guided Decoding for Three-Way Logical Question Answering
Not on the map. Why?
Abstract (arXiv)
Three-way logical question answering (QA) assigns one of True, False, or Unknown to a hypothesis H given a premise set S. We study this task as a compact compositional inference problem: predictions for H and for a mechanically negated hypothesis neg H should agree under a deterministic negation map. Despite this simple structure, large language models (LLMs) can exhibit two practical failure modes: (i) negation inconsistency, where answers to H and neg H violate the required label mapping, and (ii) epistemic Unknown, where the model abstains even when one side is entailed. We introduce CGD-PD, a lightweight, training-free test-time layer that combines neural 3-way classification, symbolic negation-consistency projection, and targeted binary entailment probes. On one validation split of FOLIO's first-order logic fields, CGD-PD improves accuracy by 4.4 points on GPT-5.2 and 6.8 points on Claude Sonnet 4.5, while reducing Unknown predictions and epistemic abstention. These results provide a controlled proof of concept that simple logical composition at inference time can help evaluate and improve LLM reasoning reliability; they do not, by themselves, establish robustness beyond this formal benchmark setting.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 15 of its 21 references that are in the map. First posted on arXiv 12 Mar 2026.
Builds on
15 of 21 references.
References · 15
- 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
- 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
- 2015A large annotated corpus for learning natural language inferencecited by 1,837
- 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,259
- 2020Beyond Accuracy: Behavioral Testing of NLP models with CheckListcited by 979
- 2017Selective Classification for Deep Neural Networkscited by 799
- 2021PICARD: Parsing Incrementally for Constrained Auto-Regressive Decoding from Language Modelscited by 402
- 2015Neural Module Networkscited by 375
- 2020ProofWriter: Generating Implications, Proofs, and Abductive Statements over Natural Languagecited by 309
- 2019Measuring Compositional Generalization: A Comprehensive Method on Realistic Datacited by 280
- 2020Transformers as Soft Reasoners over Languagecited by 265
- 2017Lexically Constrained Decoding for Sequence Generation Using Grid Beam Searchcited by 234
3 more
- 2018Fast Lexically Constrained Decoding with Dynamic Beam Allocation for Neural Machine Translationcited by 181
- 2024P-FOLIO: Evaluating and Improving Logical Reasoning with Abundant Human-Written Reasoning Chainscited by 13
- 2025Metamorphic Testing of Large Language Models for Natural Language Processingcited by 12
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026Controlled Reformulation Testing for Logical Consistency in Large Language Models
- 2026From Atomic Evidence to Logical Composition: Structured Compositional Reasoning over Compound Answer Options
- 2026Operadic consistency: a label-free signal for compositional reasoning failures in LLMs
- 2025Latent Self-Consistency for Reliable Majority-Set Selection in Short- and Long-Answer Reasoning
- 2022Measuring Compositional Consistency for Video Question Answering
- 2026PassiveQA: A Three-Action Framework for Epistemically Calibrated Question Answering via Supervised Finetuning
- 2022Faithful Reasoning Using Large Language Models
- 2024LogicBench: Towards Systematic Evaluation of Logical Reasoning Ability of Large Language Models