# Flying Pigs, FaR and Beyond: Evaluating LLM Reasoning in Counterfactual Worlds

arXiv: https://arxiv.org/abs/2505.22318 · HTML: https://mlsanity.com/paper/2505.22318/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning.
- Builds on 32 of its 67 references that are in the map.
- First posted on arXiv 28 May 2025.

## Placed in

- LLM reasoning / [Reasoning benchmarks and evaluation](https://mlsanity.com/lr/reasoning-benchmarks-and-evaluation/)

## Builds on (most cited first, 30 of 32)

- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [Tree of Thoughts: Deliberate Problem Solving with Large Language Models](https://mlsanity.com/paper/2305.10601/) (2023)
- [How Much Knowledge Can You Pack Into the Parameters of a Language Model?](https://mlsanity.com/paper/2002.08910/) (2020)
- [Adaptive Chameleon or Stubborn Sloth: Revealing the Behavior of Large Language Models in Knowledge Conflicts](https://mlsanity.com/paper/2305.13300/) (2023)
- [HippoRAG: Neurobiologically Inspired Long-Term Memory for Large Language Models](https://mlsanity.com/paper/2405.14831/) (2024)
- [Reasoning or Reciting? Exploring the Capabilities and Limitations of Language Models Through Counterfactual Tasks](https://mlsanity.com/paper/2307.02477/) (2023)
- [FOLIO: Natural Language Reasoning with First-Order Logic](https://mlsanity.com/paper/2209.00840/) (2022)
- [Survey on Factuality in Large Language Models: Knowledge, Retrieval and Domain-Specificity](https://mlsanity.com/paper/2310.07521/) (2023)
- [Language models show human-like content effects on reasoning tasks](https://mlsanity.com/paper/2207.07051/) (2022)
- [On Memorization of Large Language Models in Logical Reasoning](https://mlsanity.com/paper/2410.23123/) (2024)
- [Navigating the Grey Area: How Expressions of Uncertainty and Overconfidence Affect Language Models](https://mlsanity.com/paper/2302.13439/) (2023)
- [ConflictBank: A Benchmark for Evaluating the Influence of Knowledge Conflicts in LLM](https://mlsanity.com/paper/2408.12076/) (2024)
- [LogicBench: Towards Systematic Evaluation of Logical Reasoning Ability of Large Language Models](https://mlsanity.com/paper/2404.15522/) (2024)
- [Metacognitive Prompting Improves Understanding in Large Language Models](https://mlsanity.com/paper/2308.05342/) (2023)
- [Resolving Knowledge Conflicts in Large Language Models](https://mlsanity.com/paper/2310.00935/) (2023)
- [Using Counterfactual Tasks to Evaluate the Generality of Analogical Reasoning in Large Language Models](https://mlsanity.com/paper/2402.08955/) (2024)
- [DISCO: Distilling Counterfactuals with Large Language Models](https://mlsanity.com/paper/2212.10534/) (2022)
- [CounterBench: Evaluating and Improving Counterfactual Reasoning in Large Language Models](https://mlsanity.com/paper/2502.11008/) (2025)
- [A Systematic Comparison of Syllogistic Reasoning in Humans and Language Models](https://mlsanity.com/paper/2311.00445/) (2023)
- [General Purpose Verification for Chain of Thought Prompting](https://mlsanity.com/paper/2405.00204/) (2024)
- [Knowledge Editing in Language Models via Adapted Direct Preference Optimization](https://mlsanity.com/paper/2406.09920/) (2024)
- [A Implies B: Circuit Analysis in LLMs for Propositional Logical Reasoning](https://mlsanity.com/paper/2411.04105/) (2024)
- [A Systematic Analysis of Large Language Models as Soft Reasoners: The Case of Syllogistic Inferences](https://mlsanity.com/paper/2406.11341/) (2024)
- [Right for Right Reasons: Large Language Models for Verifiable Commonsense Knowledge Graph Question Answering](https://mlsanity.com/paper/2403.01390/) (2024)
- [Negation: A Pink Elephant in the Large Language Models' Room?](https://mlsanity.com/paper/2503.22395/) (2025)
- [Generative Evaluation of Complex Reasoning in Large Language Models](https://mlsanity.com/paper/2504.02810/) (2025)
- [ContraSolver: Self-Alignment of Language Models by Resolving Internal Preference Contradictions](https://mlsanity.com/paper/2406.08842/) (2024)
- [SaGE: Evaluating Moral Consistency in Large Language Models](https://mlsanity.com/paper/2402.13709/) (2024)
- [Untangle the KNOT: Interweaving Conflicting Knowledge and Reasoning Skills in Large Language Models](https://mlsanity.com/paper/2404.03577/) (2024)

## Built on by (most cited first, 3 of 3)

- [Know More, Know Clearer: A Meta-Cognitive Framework for Knowledge Augmentation in Large Language Models](https://mlsanity.com/paper/2602.12996/) (2026)
- [The Illusion of textitWhat If: Evaluating the Breakdown of Counterfactual Reasoning in LLMs](https://mlsanity.com/paper/2608.27953/) (2026)
- [Understanding and Mitigating Inference-Time Overreliance Using Agentic Memory](https://mlsanity.com/paper/2610.07311/) (2026)

## Abstract (arXiv)

A fundamental challenge in reasoning is navigating hypothetical, counterfactual worlds where logic may conflict with ingrained knowledge. We investigate this frontier for Large Language Models (LLMs) by asking: Can LLMs reason logically when the context contradicts their parametric knowledge? To facilitate a systematic analysis, we first introduce CounterLogic, a benchmark specifically designed to disentangle logical validity from knowledge alignment. Evaluation of 11 LLMs across six diverse reasoning datasets reveals a consistent failure: model accuracy plummets by an average of 14% in counterfactual scenarios compared to knowledge-aligned ones. We hypothesize that this gap stems not from a flaw in logical processing, but from an inability to manage the cognitive conflict between context and knowledge. Inspired by human metacognition, we propose a simple yet powerful intervention: Flag & Reason (FaR), where models are first prompted to flag potential knowledge conflicts before they reason. This metacognitive step is highly effective, narrowing the performance gap to just 7% and increasing overall accuracy by 4%. Our findings diagnose and study a critical limitation in modern LLMs' reasoning and demonstrate how metacognitive awareness can make them more robust and reliable thinkers.