How Clued up are LLMs? Evaluating Multi-Step Deductive Reasoning in a Text-Based Game Environment
On the map: LLM reasoning → Reasoning for agents, planning and code
also in 1 more
Abstract (arXiv)
Deducing whodunit proves challenging for LLM agents. In this paper, we implement a text-based multi-agent version of the classic board game Clue as a rule-based testbed for evaluating multi-step deductive reasoning, with six agents drawn from GPT-4o-mini and Gemini-2.5-Flash. We further investigate whether fine-tuning on structured logic puzzles transfers to improved in-game reasoning and gameplay. Across 18 simulated games, agents achieve only four correct wins, indicating difficulty in maintaining consistent deductive reasoning over the course of a full game. Additionally, we find that fine-tuning does not reliably improve performance and, in some cases, appears to increase reasoning volume without improving reasoning precision.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning. Builds on 20 of its 26 references that are in the map. First posted on arXiv 17 Mar 2026.
Builds on
20 of 26 references.
Closest ancestors · 3
Same direction, newest first
- 2025DyFlow: Dynamic Workflow Framework for Agentic Reasoning30 Sep
- 2025Exploring the Necessity of Reasoning in LLM-based Agent Scenarios14 Mar
- 2024GAMEBoT: Transparent Assessment of LLM Reasoning in Games18 Dec
Other references · 17
- 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
- 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,655
- 2023AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversationcited by 1,906
- 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
- 2020ReClor: A Reading Comprehension Dataset Requiring Logical Reasoningcited by 211
- 2023Navigate through Enigmatic Labyrinth A Survey of Chain of Thought Reasoning: Advances, Frontiers and Futurecited by 184
- 2023Language Agents with Reinforcement Learning for Strategic Play in the Werewolf Gamecited by 148
- 2024Evaluating Mathematical Reasoning Beyond Accuracycited by 69
- 2023MR-GSM8K: A Meta-Reasoning Benchmark for Large Language Model Evaluationcited by 57
- 2024Multi-LogiEval: Towards Evaluating Multi-Step Logical Reasoning Ability of Large Language Modelscited by 38
- 2024On the Impact of Fine-Tuning on Chain-of-Thought Reasoningcited by 32
- 2025The Curse of CoT: On the Limitations of Chain-of-Thought in In-Context Learningcited by 30
5 more
- 2024AMONGAGENTS: Evaluating Large Language Models in the Interactive Text-Based Social Deduction Gamecited by 26
- 2023Probing the Multi-turn Planning Capabilities of LLMs via 20 Question Gamescited by 24
- 2025PokerBench: Training Large Language Models to become Professional Poker Playerscited by 18
- 2024Practical Considerations for Agentic LLM Systemscited by 8
- 2025Beyond Correctness: Rewarding Faithful Reasoning in Retrieval-Augmented Generationcited by 6
Built on by
Citing papers per year: 2026: 1.
1 paper.
- 2026Clueing up LLMs with Tool-Augmented Deductive Reasoningcited by 0
- 2026Clueing up LLMs with Tool-Augmented Deductive Reasoning16 Sep 2026
Similar, not cited
Close in wording, no citation either way
- 2025Baba is LLM: Reasoning in a Game with Dynamic Rules
- 2024GTBench: Uncovering the Strategic Reasoning Limitations of LLMs via Game-Theoretic Evaluations
- 2024LogicGame: Benchmarking Rule-Based Reasoning Abilities of Large Language Models
- 2024GameBench: Evaluating Strategic Reasoning Abilities of LLM Agents
- 2025TurnaboutLLM: A Deductive Reasoning Benchmark from Detective Games
- 2026Reasoning Capabilities of Large Language Models. Lessons Learned from General Game Playing
- 2024GameArena: Evaluating LLM Reasoning through Live Computer Games
- 2024TMGBench: A Systematic Game Benchmark for Evaluating Strategic Reasoning Abilities of LLMs