mlsanityData to 14 Oct 2026

From Atomic Evidence to Logical Composition: Structured Compositional Reasoning over Compound Answer Options

arXiv 2608.12836 ↗First posted 13 Aug 2026cs.CL cs.AIcited by 0

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

Large language models often fail when answer options require combining atomic judgments under explicit logical operators, even when they judge the individual atoms correctly. We study compound options connected by AND, OR, and NEITHER/NOR, introducing a framework that decomposes each option into atomic answers and scores contrastive hypotheses about each one, so the model never sees a compound option. An operator-constrained integer linear program then composes the calibrated scores into a single prediction. We evaluate on LOGICAL-COMMONSENSEQA and introduce LOGICAL-SATA, a reading-comprehension benchmark derived from SATA-Bench. Our framework improves Macro-F1 from 48.3 to 77.0 on the human-validated LOGICAL-COMMONSENSEQA split and from 47.0 to 75.6 on LOGICAL-SATA, with the largest gains on NEITHER/NOR.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 6 of its 59 references that are in the map. First posted on arXiv 13 Aug 2026.

Builds on

6 of 59 references.

References · 6

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  3. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  4. 2022Decomposed Prompting: A Modular Approach for Solving Complex Taskscited by 558
  5. 2020ReClor: A Reading Comprehension Dataset Requiring Logical Reasoningcited by 217
  6. 2025SATA-BENCH: Select All That Apply Benchmark for Multiple Choice Questionscited by 9

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026LOGICAL-COMMONSENSEQA: A Benchmark for Logical Commonsense Reasoning
  2. 2026Operads for compositional reasoning in LLMs
  3. 2024Divide and Translate: Compositional First-Order Logic Translation and Verification for Complex Logical Reasoning
  4. 2026Quantifying Consistency in LLM Logical Reasoning via Structural Uncertainty
  5. 2026Benchmarking LLM Competence on Logical Inference over Probability Operators
  6. 2024Plausibly Problematic Questions in Multiple-Choice Benchmarks for Commonsense Reasoning
  7. 2026Operadic consistency: a label-free signal for compositional reasoning failures in LLMs
  8. 2026Evaluating Robustness of Reasoning Models on Parameterized Logical Problems