# GuessingGame: Measuring the Informativeness of Open-Ended Questions in Large Language Models

arXiv: https://arxiv.org/abs/2509.19593 · HTML: https://mlsanity.com/paper/2509.19593/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 48 references that are in the map.
- First posted on arXiv 23 Sep 2025.

## Builds on (most cited first, 24 of 24)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [A Survey on Large Language Model based Autonomous Agents](https://mlsanity.com/paper/2308.11432/) (2023)
- [ConceptNet 5.5: An Open Multilingual Graph of General Knowledge](https://mlsanity.com/paper/1612.03975/) (2016)
- [Large Language Models as Commonsense Knowledge for Large-Scale Task Planning](https://mlsanity.com/paper/2305.14078/) (2023)
- [MediQ: Question-Asking LLMs and a Benchmark for Reliable Interactive Clinical Reasoning](https://mlsanity.com/paper/2406.00922/) (2024)
- [Asking Clarifying Questions in Open-Domain Information-Seeking Conversations](https://mlsanity.com/paper/1907.06554/) (2019)
- [Rephrase and Respond: Let Large Language Models Ask Better Questions for Themselves](https://mlsanity.com/paper/2311.04205/) (2023)
- [On Verbalized Confidence Scores for LLMs](https://mlsanity.com/paper/2412.14737/) (2024)
- [PROST: Physical Reasoning of Objects through Space and Time](https://mlsanity.com/paper/2106.03634/) (2021)
- [NEWTON: Are Large Language Models Capable of Physical Reasoning?](https://mlsanity.com/paper/2310.07018/) (2023)
- [Standards for Belief Representations in LLMs](https://mlsanity.com/paper/2405.21030/) (2024)
- [Improving Hypernymy Detection with an Integrated Path-based and Distributional Method](https://mlsanity.com/paper/1603.06076/) (2016)
- [Verb Physics: Relative Physical Knowledge of Actions and Objects](https://mlsanity.com/paper/1706.03799/) (2017)
- [Probing the Multi-turn Planning Capabilities of LLMs via 20 Question Games](https://mlsanity.com/paper/2310.01468/) (2023)
- [Learning to Ask Informative Questions: Enhancing LLMs with Preference Optimization and Expected Information Gain](https://mlsanity.com/paper/2406.17453/) (2024)
- [Loose LIPS Sink Ships: Asking Questions in Battleship with Language-Informed Program Sampling](https://mlsanity.com/paper/2402.19471/) (2024)
- [Acquiring Common Sense Spatial Knowledge through Implicit Spatial Templates](https://mlsanity.com/paper/1711.06821/) (2017)
- [Do Large Language Models have Problem-Solving Capability under Incomplete Information Scenarios?](https://mlsanity.com/paper/2409.14762/) (2024)
- [Asking More Informative Questions for Grounded Retrieval](https://mlsanity.com/paper/2311.08584/) (2023)
- [Chatbots as Problem Solvers: Playing Twenty Questions with Role Reversals](https://mlsanity.com/paper/2301.01743/) (2023)
- [TEXT2AFFORD: Probing Object Affordance Prediction abilities of Language Models solely from Text](https://mlsanity.com/paper/2402.12881/) (2024)
- [Twenty (simple) questions](https://mlsanity.com/paper/1611.01655/) (2016)

## Built on by (most cited first, 1 of 1)

- [Benchmarking Agents in Insurance Underwriting Environments](https://mlsanity.com/paper/2602.00456/) (2026)

## Abstract (arXiv)

We introduce GuessingGame, a protocol for evaluating large language models (LLMs) as strategic question-askers in open-ended, open-domain settings. A Guesser LLM identifies a hidden object by posing free-form questions to an Oracle without predefined choices or candidate lists. To measure question quality, we propose two information gain (IG) metrics: a Bayesian method that tracks belief updates over semantic concepts using LLM-scored relevance, and an entropy-based method that filters candidates via ConceptNet. Both metrics are model-agnostic and support post hoc analysis. Across 858 games with multiple models and prompting strategies, higher IG strongly predicts efficiency: a one-standard-deviation IG increase reduces expected game length by 43%. Prompting constraints guided by IG, such as enforcing question diversity, enable weaker models to significantly improve performance. These results show that question-asking in LLMs is both measurable and improvable, and crucial for interactive reasoning.