mlsanityData to 14 Oct 2026

GuessingGame: Measuring the Informativeness of Open-Ended Questions in Large Language Models

arXiv 2509.19593 ↗First posted 23 Sep 2025cs.CL cs.AIcited by 1

Not on the map. Why?

Abstract (arXiv)

We introduce GuessingGame, a protocol for evaluating large language models (LLMs) as strategic question-askers in open-ended, open-domain settings. A Guesser LLM identifies a hidden object by posing free-form questions to an Oracle without predefined choices or candidate lists. To measure question quality, we propose two information gain (IG) metrics: a Bayesian method that tracks belief updates over semantic concepts using LLM-scored relevance, and an entropy-based method that filters candidates via ConceptNet. Both metrics are model-agnostic and support post hoc analysis. Across 858 games with multiple models and prompting strategies, higher IG strongly predicts efficiency: a one-standard-deviation IG increase reduces expected game length by 43%. Prompting constraints guided by IG, such as enforcing question diversity, enable weaker models to significantly improve performance. These results show that question-asking in LLMs is both measurable and improvable, and crucial for interactive reasoning.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 24 of its 48 references that are in the map. First posted on arXiv 23 Sep 2025.

Builds on

24 of 48 references.

References · 24

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  3. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  4. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,242
  5. 2016ConceptNet 5.5: An Open Multilingual Graph of General Knowledgecited by 1,067
  6. 2023Large Language Models as Commonsense Knowledge for Large-Scale Task Planningcited by 231
  7. 2024MediQ: Question-Asking LLMs and a Benchmark for Reliable Interactive Clinical Reasoningcited by 154
  8. 2019Asking Clarifying Questions in Open-Domain Information-Seeking Conversationscited by 139
  9. 2023Rephrase and Respond: Let Large Language Models Ask Better Questions for Themselvescited by 123
  10. 2024On Verbalized Confidence Scores for LLMscited by 83
  11. 2021PROST: Physical Reasoning of Objects through Space and Timecited by 48
  12. 2023NEWTON: Are Large Language Models Capable of Physical Reasoning?cited by 45
12 more
  1. 2024Standards for Belief Representations in LLMscited by 30
  2. 2016Improving Hypernymy Detection with an Integrated Path-based and Distributional Methodcited by 27
  3. 2017Verb Physics: Relative Physical Knowledge of Actions and Objectscited by 26
  4. 2023Probing the Multi-turn Planning Capabilities of LLMs via 20 Question Gamescited by 24
  5. 2024Learning to Ask Informative Questions: Enhancing LLMs with Preference Optimization and Expected Information Gaincited by 21
  6. 2024Loose LIPS Sink Ships: Asking Questions in Battleship with Language-Informed Program Samplingcited by 12
  7. 2017Acquiring Common Sense Spatial Knowledge through Implicit Spatial Templatescited by 10
  8. 2024Do Large Language Models have Problem-Solving Capability under Incomplete Information Scenarios?cited by 7
  9. 2023Asking More Informative Questions for Grounded Retrievalcited by 6
  10. 2023Chatbots as Problem Solvers: Playing Twenty Questions with Role Reversalscited by 3
  11. 2024TEXT2AFFORD: Probing Object Affordance Prediction abilities of Language Models solely from Textcited by 3
  12. 2016Twenty (simple) questionscited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Benchmarking Agents in Insurance Underwriting Environmentscited by 0