# Same Formulas, Different Semantics: Do Language Models Follow Modal Logic Specifications?

arXiv: https://arxiv.org/abs/2608.05097 · HTML: https://mlsanity.com/paper/2608.05097/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 5 of its 16 references that are in the map.
- First posted on arXiv 5 Aug 2026.

## Builds on (most cited first, 5 of 5)

- [ProofWriter: Generating Implications, Proofs, and Abductive Statements over Natural Language](https://mlsanity.com/paper/2012.13048/) (2020)
- [MindGames: Targeting Theory of Mind in Large Language Models with Dynamic Epistemic Modal Logic](https://mlsanity.com/paper/2305.03353/) (2023)
- [Extensional Higher-Order Paramodulation in Leo-III](https://mlsanity.com/paper/1907.11501/) (2019)
- [SpaceNLI: Evaluating the Consistency of Predicting Inferences in Space](https://mlsanity.com/paper/2307.02269/) (2023)
- [Solving Quantified Modal Logic Problems by Translation to Classical Logics](https://mlsanity.com/paper/2212.09570/) (2022)

## Abstract (arXiv)

Reasoning about necessity and possibility depends on assumptions about accessibility between worlds and about which objects exist at each one. The same inference may therefore hold under one modal system and fail under another. Evaluating language models on such problems requires testing whether their judgments follow the stated semantics rather than a familiar logic. We construct paired modal problems with identical premises and conjecture but different frame or domain conditions; automated reasoning verifies opposite labels. A balanced core prevents the semantic condition alone from revealing the answer. On this core, four of five recent models perform below the condition-only baseline under direct prompting. Yet enabling reasoning mode raises DeepSeek V4 Flash from 4.4% to 88.1% on unchanged prompts. Following stipulated modal semantics thus depends strongly on inference mode as well as model identity. When frame conditions are omitted, models often agree but fit different familiar logics best. We release the formulas, oracle artifacts, countermodels, and responses.