# Disentangling Logic: The Role of Context in Large Language Model Reasoning Capabilities

arXiv: https://arxiv.org/abs/2406.02787 · HTML: https://mlsanity.com/paper/2406.02787/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning.
- Builds on 20 of its 35 references that are in the map.
- First posted on arXiv 4 Jun 2024.

## Placed in

- LLM reasoning / [Reasoning benchmarks and evaluation](https://mlsanity.com/lr/reasoning-benchmarks-and-evaluation/)

## Builds on (most cited first, 20 of 20)

- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [Qwen Technical Report](https://mlsanity.com/paper/2309.16609/) (2023)
- [Scaling Instruction-Finetuned Language Models](https://mlsanity.com/paper/2210.11416/) (2022)
- [Emergent Abilities of Large Language Models](https://mlsanity.com/paper/2206.07682/) (2022)
- [Yi: Open Foundation Models by 01.AI](https://mlsanity.com/paper/2403.04652/) (2024)
- [Logic-LM: Empowering Large Language Models with Symbolic Solvers for Faithful Logical Reasoning](https://mlsanity.com/paper/2305.12295/) (2023)
- [PlanBench: An Extensible Benchmark for Evaluating Large Language Models on Planning and Reasoning about Change](https://mlsanity.com/paper/2206.10498/) (2022)
- [Language Models Are Greedy Reasoners: A Systematic Formal Analysis of Chain-of-Thought](https://mlsanity.com/paper/2210.01240/) (2022)
- [Language models show human-like content effects on reasoning tasks](https://mlsanity.com/paper/2207.07051/) (2022)
- [War and Peace (WarAgent): Large Language Model-based Multi-Agent Simulation of World Wars](https://mlsanity.com/paper/2311.17227/) (2023)
- [NPHardEval: Dynamic Benchmark on Reasoning Ability of Large Language Models via Complexity Classes](https://mlsanity.com/paper/2312.14890/) (2023)
- [ARB: Advanced Reasoning Benchmark for Large Language Models](https://mlsanity.com/paper/2307.13692/) (2023)
- [IsoBench: Benchmarking Multimodal Foundation Models on Isomorphic Representations](https://mlsanity.com/paper/2404.01266/) (2024)
- [BattleAgent: Multi-modal Dynamic Emulation on Historical Battles to Complement Historical Analysis](https://mlsanity.com/paper/2404.15532/) (2024)
- [Health-LLM: Personalized Retrieval-Augmented Disease Prediction System](https://mlsanity.com/paper/2402.00746/) (2024)
- [What if LLMs Have Different World Views: Simulating Alien Civilizations with LLM-based Agents](https://mlsanity.com/paper/2402.13184/) (2024)
- [Reason from Fallacy: Enhancing Large Language Models' Logical Reasoning through Logical Fallacy Understanding](https://mlsanity.com/paper/2404.04293/) (2024)
- [DataChat: Prototyping a Conversational Agent for Dataset Search and Visualization](https://mlsanity.com/paper/2305.18358/) (2023)

## Built on by (most cited first, 22 of 22)

- [To CoT or not to CoT? Chain-of-thought helps mainly on math and symbolic reasoning](https://mlsanity.com/paper/2409.12183/) (2024)
- [AgentReview: Exploring Peer Review Dynamics with LLM Agents](https://mlsanity.com/paper/2406.12708/) (2024)
- [OmniRouter: Budget and Performance Controllable Multi-LLM Routing](https://mlsanity.com/paper/2502.20576/) (2025)
- [RuleArena: A Benchmark for Rule-Guided Reasoning with LLMs in Real-World Scenarios](https://mlsanity.com/paper/2412.08972/) (2024)
- [Simulated patient systems powered by large language model-based AI agents offer potential for transforming medical education](https://mlsanity.com/paper/2409.18924/) (2024)
- [A Survey on Human-Centric LLMs](https://mlsanity.com/paper/2411.14491/) (2024)
- [From Pixels to Places: A Systematic Benchmark for Evaluating Image Geolocalization Ability in Large Language Models](https://mlsanity.com/paper/2508.01608/) (2025)
- [Invisible Prompts, Visible Threats: Malicious Font Injection in External Resources for Large Language Models](https://mlsanity.com/paper/2505.16957/) (2025)
- [InductionBench: LLMs Fail in the Simplest Complexity Class](https://mlsanity.com/paper/2502.15823/) (2025)
- [Cache Mechanism for Agent RAG Systems](https://mlsanity.com/paper/2511.02919/) (2025)
- [Token Signature: Predicting Chain-of-Thought Gains with Token Decoding Feature in Large Language Models](https://mlsanity.com/paper/2506.06008/) (2025)
- [Meet Dynamic Individual Preferences: Resolving Conflicting Human Value with Paired Fine-Tuning](https://mlsanity.com/paper/2604.12479/) (2026)
- [Fine-Tuning is Subgraph Search: A New Lens on Learning Dynamics](https://mlsanity.com/paper/2502.06106/) (2025)
- [Socrates or Smartypants: Testing Logic Reasoning Capabilities of Large Language Models with Logic Programming-based Test Oracles](https://mlsanity.com/paper/2504.12312/) (2025)
- [AlgoSimBench: Identifying Algorithmically Similar Problems for Competitive Programming](https://mlsanity.com/paper/2507.15378/) (2025)
- [When Do LLMs Reason? A Dynamical Systems View via Entropy Phase Transitions](https://mlsanity.com/paper/2605.22873/) (2026)
- [TypedThinker: Diversify Large Language Model Reasoning with Typed Thinking](https://mlsanity.com/paper/2410.01952/) (2024)
- [Individual Turing Test: A Case Study of LLM-based Simulation Using Longitudinal Personal Data](https://mlsanity.com/paper/2603.01289/) (2026)
- [Reforming the Mechanism: Editing Reasoning Patterns in LLMs with Circuit Reshaping](https://mlsanity.com/paper/2603.06923/) (2026)
- [On the Thinking-Language Modeling Gap in Large Language Models](https://mlsanity.com/paper/2505.12896/) (2025)
- [Dual-Process Scaffold Reasoning for Enhancing LLM Code Debugging](https://mlsanity.com/paper/2511.08052/) (2025)
- [Towards Precision Therapy in Hepatocellular Carcinoma: A Clinical-Reasoning LLM for Risk Stratification and Treatment Guidance](https://mlsanity.com/paper/2607.08602/) (2026)

## Abstract (arXiv)

This study intends to systematically disentangle pure logic reasoning and text understanding by investigating the contrast across abstract and contextualized logical problems from a comprehensive set of domains. We explore whether LLMs demonstrate genuine reasoning capabilities across various domains when the underlying logical structure remains constant. We focus on two main questions (1) Can abstract logical problems alone accurately benchmark an LLM's reasoning ability in real-world scenarios, disentangled from contextual support in practical settings? (2) Does fine-tuning LLMs on abstract logic problem generalize to contextualized logic problems and vice versa? To investigate these questions, we focus on standard propositional logic, specifically propositional deductive and abductive logic reasoning. In particular, we construct instantiated datasets for deductive and abductive reasoning with 4 levels of difficulty, encompassing 12 distinct categories or domains based on the categorization of Wikipedia. Our experiments aim to provide insights into disentangling context in logical reasoning and the true reasoning capabilities of LLMs and their generalization potential. The code and dataset are available at: https://github.com/agiresearch/ContextHub.