# Controlled Reformulation Testing for Logical Consistency in Large Language Models

arXiv: https://arxiv.org/abs/2607.14528 · HTML: https://mlsanity.com/paper/2607.14528/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 5 of its 6 references that are in the map.
- First posted on arXiv 16 Jul 2026.

## Builds on (most cited first, 5 of 5)

- [Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting](https://mlsanity.com/paper/2305.04388/) (2023)
- [FOLIO: Natural Language Reasoning with First-Order Logic](https://mlsanity.com/paper/2209.00840/) (2022)
- [LogicBench: Towards Systematic Evaluation of Logical Reasoning Ability of Large Language Models](https://mlsanity.com/paper/2404.15522/) (2024)
- [Existing LLMs Are Not Self-Consistent For Simple Tasks](https://mlsanity.com/paper/2506.18781/) (2025)
- [Finding the Cracks: Improving LLMs Reasoning with Paraphrastic Probing and Consistency Verification](https://mlsanity.com/paper/2602.11361/) (2026)

## Built on by (most cited first, 1 of 1)

- [Hob-VL: A Benchmark for Visually Grounded Boolean Reasoning](https://mlsanity.com/paper/2610.01605/) (2026)

## Abstract (arXiv)

Large language models (LLMs) frequently contradict themselves when the surface form of a logically equivalent question changes. We present a benchmark of 350 question families (1,750 total questions) for Controlled Reformulation Testing (CRTBench) to evaluate logical invariance. In this benchmark, we investigate LLMs' ability to maintain consistent answers across controlled reformulations, which include contrapositive rewriting, double negation, negation flipping, and passive voice. We evaluate several frontier LLMs and observe an accuracy-consistency gap where GPT-5.4-mini achieves 98.9% base accuracy but only 60.3% family-level consistency, while reasoning-optimized o4-mini achieves 96.9% consistency. From our experiments, we observe that failures cluster around logically nontrivial transformations such as contrapositive rewriting (72.4% for GPT-5.4-mini) and double negation (84.6%), while surface-level rephrasing remains robust (94-100%). Increasing reasoning effort improves GPT-5.4-mini to 85.4% consistency, but leaves GPT-5.4 unchanged overall because gains on nested negation are offset by failures on quantifier families. These results show that accuracy alone is not enough for evaluating logical reasoning in LLMs.