# What to Ask Next? Probing the Imaginative Reasoning of LLMs with TurtleSoup Puzzles

arXiv: https://arxiv.org/abs/2508.10358 · HTML: https://mlsanity.com/paper/2508.10358/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning.
- Builds on 10 of its 46 references that are in the map.
- First posted on arXiv 14 Aug 2025.

## Placed in

- LLM reasoning / [Reasoning benchmarks and evaluation](https://mlsanity.com/lr/reasoning-benchmarks-and-evaluation/)

## Builds on (most cited first, 10 of 10)

- [Can We Trust AI Benchmarks? An Interdisciplinary Review of Current Issues in AI Evaluation](https://mlsanity.com/paper/2502.06559/) (2025)
- [The Traitors: Deception and Trust in Multi-Agent Language Model Simulations](https://mlsanity.com/paper/2505.12923/) (2025)
- [What Makes a Good Story and How Can We Measure It? A Comprehensive Survey of Story Evaluation](https://mlsanity.com/paper/2408.14622/) (2024)
- [MTR-Bench: A Comprehensive Benchmark for Multi-Turn Reasoning Evaluation](https://mlsanity.com/paper/2505.17123/) (2025)
- [RPGBENCH: Evaluating Large Language Models as Role-Playing Game Engines](https://mlsanity.com/paper/2502.00595/) (2025)
- [Leveraging Word Guessing Games to Assess the Intelligence of Large Language Models](https://mlsanity.com/paper/2310.20499/) (2023)
- [TurtleBench: Evaluating Top Language Models via Real-World Yes/No Puzzles](https://mlsanity.com/paper/2410.05262/) (2024)
- [MIRAGE: Exploring How Large Language Models Perform in Complex Social Interactive Environments](https://mlsanity.com/paper/2501.01652/) (2025)
- [Exploring Large Language Models for Word Games:Who is the Spy?](https://mlsanity.com/paper/2503.15235/) (2025)
- [Seeing Beyond Words: MatVQA for Challenging Visual-Scientific Reasoning in Materials Science](https://mlsanity.com/paper/2505.18319/) (2025)

## Abstract (arXiv)

We investigate the capacity of Large Language Models (LLMs) for imaginative reasoning--the proactive construction, testing, and revision of hypotheses in information-sparse environments. Existing benchmarks, often static or focused on social deduction, fail to capture the dynamic, exploratory nature of this reasoning process. To address this gap, we introduce a comprehensive research framework based on the classic "Turtle Soup" game, integrating a benchmark, an agent, and an evaluation protocol. We present TurtleSoup-Bench, the first large-scale, bilingual, interactive benchmark for imaginative reasoning, comprising 800 turtle soup puzzles sourced from both the Internet and expert authors. We also propose Mosaic-Agent, a novel agent designed to assess LLMs' performance in this setting. To evaluate reasoning quality, we develop a multi-dimensional protocol measuring logical consistency, detail completion, and conclusion alignment. Experiments with leading LLMs reveal clear capability limits, common failure patterns, and a significant performance gap compared to humans. Our work offers new insights into LLMs' imaginative reasoning and establishes a foundation for future research on exploratory agent behavior.