mlsanityData to 14 Oct 2026

What to Ask Next? Probing the Imaginative Reasoning of LLMs with TurtleSoup Puzzles

arXiv 2508.10358 ↗First posted 14 Aug 2025cs.AIcited by 0

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

We investigate the capacity of Large Language Models (LLMs) for imaginative reasoning--the proactive construction, testing, and revision of hypotheses in information-sparse environments. Existing benchmarks, often static or focused on social deduction, fail to capture the dynamic, exploratory nature of this reasoning process. To address this gap, we introduce a comprehensive research framework based on the classic "Turtle Soup" game, integrating a benchmark, an agent, and an evaluation protocol. We present TurtleSoup-Bench, the first large-scale, bilingual, interactive benchmark for imaginative reasoning, comprising 800 turtle soup puzzles sourced from both the Internet and expert authors. We also propose Mosaic-Agent, a novel agent designed to assess LLMs' performance in this setting. To evaluate reasoning quality, we develop a multi-dimensional protocol measuring logical consistency, detail completion, and conclusion alignment. Experiments with leading LLMs reveal clear capability limits, common failure patterns, and a significant performance gap compared to humans. Our work offers new insights into LLMs' imaginative reasoning and establishes a foundation for future research on exploratory agent behavior.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 10 of its 46 references that are in the map. First posted on arXiv 14 Aug 2025.

Builds on

10 of 46 references.

Closest ancestors · 1

Same direction, newest first

  1. 2025MTR-Bench: A Comprehensive Benchmark for Multi-Turn Reasoning Evaluation21 May

Other references · 9

  1. 2025Can We Trust AI Benchmarks? An Interdisciplinary Review of Current Issues in AI Evaluationcited by 60
  2. 2025The Traitors: Deception and Trust in Multi-Agent Language Model Simulationscited by 34
  3. 2024What Makes a Good Story and How Can We Measure It? A Comprehensive Survey of Story Evaluationcited by 17
  4. 2025RPGBENCH: Evaluating Large Language Models as Role-Playing Game Enginescited by 12
  5. 2023Leveraging Word Guessing Games to Assess the Intelligence of Large Language Modelscited by 8
  6. 2024TurtleBench: Evaluating Top Language Models via Real-World Yes/No Puzzlescited by 7
  7. 2025MIRAGE: Exploring How Large Language Models Perform in Complex Social Interactive Environmentscited by 6
  8. 2025Exploring Large Language Models for Word Games:Who is the Spy?cited by 6
  9. 2025Seeing Beyond Words: MatVQA for Challenging Visual-Scientific Reasoning in Materials Sciencecited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Multi-Turn Puzzles: Evaluating Interactive Reasoning and Strategic Dialogue in LLMs
  2. 2025Reasoning LLMs are Wandering Solution Explorers
  3. 2024Puzzle Solving using Reasoning of Large Language Models: A Survey
  4. 2026The Periodic Table of LLM Reasoning: A Structured Survey of Reasoning Paradigms, Methods, and Failure Modes
  5. 2025RiddleBench: A New Generative Reasoning Benchmark for LLMs
  6. 2025A Survey of Frontiers in LLM Reasoning: Inference Scaling, Learning to Reason, and Agentic Systems
  7. 2023Probing the Multi-turn Planning Capabilities of LLMs via 20 Question Games
  8. 2023Igniting Language Intelligence: The Hitchhiker's Guide From Chain-of-Thought Reasoning to Language Agents