Data to 14 Oct 2026

QuestBench: Can LLMs ask the right question to acquire information in reasoning tasks?

arXiv 2503.22674 ↗First posted 28 Mar 2025cs.AI cs.CL cs.LGcited by 63

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

Large language models (LLMs) have shown impressive performance on reasoning benchmarks like math and logic. While many works have largely assumed well-defined tasks, real-world queries are often underspecified and only solvable by acquiring missing information. We formalize this information-gathering problem as a constraint satisfaction problem (CSP) with missing variable assignments. Using a special case where only one necessary variable assignment is missing, we can evaluate an LLM's ability to identify the minimal necessary question to ask. We present QuestBench, a set of underspecified reasoning tasks solvable by asking at most one question, which includes: (1) Logic-Q: logical reasoning tasks with one missing proposition, (2) Planning-Q: PDDL planning problems with partially-observed initial states, (3) GSM-Q: human-annotated grade school math problems with one unknown variable, and (4) GSME-Q: equation-based version of GSM-Q. The LLM must select the correct clarification question from multiple options. While current models excel at GSM-Q and GSME-Q, they achieve only 40-50% accuracy on Logic-Q and Planning-Q. Analysis shows that the ability to solve well-specified reasoning problems is not sufficient for success on our benchmark: models struggle to identify the right question even when they can solve the fully specified version. This highlights the need for specifically optimizing models' information acquisition capabilities.

Key facts. Cited by 63 papers in the map (data through 14 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 48 of its 91 references that are in the map. First posted on arXiv 28 Mar 2025.

Builds on

48 of 91 references.

Closest ancestors · 2

Same direction, newest first

Other references · 46

2021Training Verifiers to Solve Math Word Problemscited by 10,109
2021Evaluating Large Language Models Trained on Codecited by 8,320
2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
2021Program Synthesis with Large Language Modelscited by 3,637
2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
2024Gemma: Open Models Based on Gemini Research and Technologycited by 916
2017Deep Bayesian Active Learning with Image Datacited by 819
2023Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generationcited by 819
2018MultiWOZ -- A Large-Scale Multi-Domain Wizard-of-Oz Dataset for Task-Oriented Dialogue Modellingcited by 766
2020A Survey of Deep Active Learningcited by 554
2019Towards Scalable Multi-domain Conversational Agents: The Schema-Guided Dialogue Datasetcited by 434
2020AmbigQA: Answering Ambiguous Open-domain Questionscited by 390

34 more

2021Dealing with Disagreements: Looking Beyond the Majority Vote in Subjective Annotationscited by 314
2024Chain of Thought Empowers Transformers to Solve Inherently Serial Problemscited by 260
2017Max-value Entropy Search for Efficient Bayesian Optimizationcited by 206
2024GSM-Plus: A Comprehensive Benchmark for Evaluating the Robustness of LLMs as Mathematical Problem Solverscited by 158
2023Generalized Planning in PDDL Domains with Pretrained Large Language Modelscited by 139
2024Taming Overconfidence in LLMs: Reward Calibration in RLHFcited by 112
2024STaR-GATE: Teaching Language Models to Ask Clarifying Questionscited by 104
2023Clarify When Necessary: Resolving Ambiguity Through Interaction with LMscited by 98
2023Eliciting Human Preferences with Language Modelscited by 92
2022Towards Learning Universal Hyperparameter Optimizers with Transformerscited by 81
2024Premise Order Matters in Reasoning with Large Language Modelscited by 72
2022CLAM: Selective Clarification for Ambiguous Questions with Generative Language Modelscited by 69
2021Pre-trained Gaussian Processes for Bayesian Optimizationcited by 63
2024Ask-before-Plan: Proactive Language Agents for Real-World Planningcited by 60
2023The Challenge of Using LLMs to Simulate Human Behavior: A Causal Inference Perspectivecited by 59
2023Grammar Prompting for Domain-Specific Language Generation with Large Language Modelscited by 58
2024CLAMBER: A Benchmark of Identifying and Clarifying Ambiguous Information Needs in Large Language Modelscited by 54
2023Everyone's Voice Matters: Quantifying Annotation Disagreement Using Demographic Informationcited by 52
2020Learning compositional models of robot skills for task and motion planningcited by 48
2024Uncertainty of Thoughts: Uncertainty-Aware Planning Enhances Information Seeking in Large Language Modelscited by 39
2024Bayesian Preference Elicitation with Language Modelscited by 34
2023Active Preference Inference using Language Models and Probabilistic Reasoningcited by 29
2018Active model learning and diverse action sampling for task and motion planningcited by 28
2018Practical Transfer Learning for Bayesian Optimizationcited by 26
2024Proactive Agents for Multi-Turn Text-to-Image Generation Under Uncertaintycited by 25
2024ChatShop: Interactive Information Seeking with Language Agentscited by 24
2024Trust the PRoC3S: Solving Long-Horizon Robotics Problems with LLMs and Constraint Satisfactioncited by 19
2024Partially Observable Task and Motion Planning with Uncertainty and Risk Awarenesscited by 17
2024Certainly Uncertain: A Benchmark and Metric for Multimodal Epistemic and Aleatoric Awarenesscited by 14
2024Loose LIPS Sink Ships: Asking Questions in Battleship with Language-Informed Program Samplingcited by 12
2024Empowering Language Models with Active Inquiry for Deeper Understandingcited by 11
2023Transfer Learning for Bayesian Optimization on Heterogeneous Search Spacescited by 9
2016Focused Model-Learning and Planning for Non-Gaussian Continuous State-Action Systemscited by 5
2023Large Language Models Should Ask Clarifying Questions to Increase Confidence in Generated Codecited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 14; 2026: 49.

63 papers.

Sort:

By area:

2025Context Length Alone Hurts LLM Performance Despite Perfect Retrievalcited by 127
2026HiL-Bench (Human-in-Loop Benchmark): Do Agents Know When to Ask for Help?cited by 27
2025BED-LLM: Intelligent Information Gathering with LLMs and Bayesian Experimental Designcited by 24
2026Calibrate-Then-Act: Cost-Aware Exploration in LLM Agentscited by 17
2025Evaluating the Goal-Directedness of Large Language Modelscited by 16
2026FIRE-Bench: Evaluating AI Agents on the Rediscovery of Scientific Insightscited by 16
2026When and What to Ask: AskBench and Rubric-Guided RLVR for LLM Clarificationcited by 16
2025Privacy Reasoning in Ambiguous Contextscited by 11
2025Shoot First, Ask Questions Later? Building Rational Agents that Explore and Act Like Peoplecited by 9
2026RealFin: How Well Do LLMs Reason About Finance When Users Leave Things Unsaid?cited by 9
2025Can LLMs Estimate Student Struggles? Human-AI Difficulty Alignment with Proficiency Simulation for Item Difficulty Predictioncited by 7
2025Towards Evaluating Proactive Risk Awareness of Multimodal Language Modelscited by 6

38 more

2025Mitigating Lost in Multi-turn Conversation via Curriculum RL with Verifiable Accuracy and Abstention Rewardscited by 6
2025Completion neq Collaboration: Scaling Collaborative Effort with Agentscited by 6
2026LHAW: Controllable Underspecification for Long-Horizon Taskscited by 6
2025The Curious Language Model: Strategic Test-Time Information Acquisitioncited by 5
2025Are Language Models Efficient Reasoners? A Perspective from Logic Programmingcited by 4
2026The Yes-Man Syndrome: Benchmarking Abstention in Embodied Robotic Agentscited by 4
2026Cold-Start Personalization via Training-Free Priors from Structured World Modelscited by 3
2026kRAIG: A Natural Language-Driven Agent for Automated DataOps Pipeline Generationcited by 3
2026Same Evidence, Different Answers: Canonical-Context On-Policy Distillation for Multi-Turn Language Modelscited by 3
2026LLMs Struggle to Measure What Distinguishes Students of Different Proficiency Levels: A Study of Item Discrimination in Reading Comprehension Assessmentcited by 3
2025The Riddle of Reflection: Evaluating Reasoning and Self-Awareness in Multilingual LLMs using Indian Riddlescited by 2
2026What Users Leave Unsaid: Under-Specified Queries Limit Vision-Language Modelscited by 2
2026Interpreting and Controlling Model Behavior via Constitutions for Atomic Concept Editscited by 2
2026The Anatomy of Uncertainty in LLMscited by 2
2026SCICONVBENCH: Benchmarking LLMs on Multi-Turn Clarification for Task Formulation in Computational Sciencecited by 2
2025Beyond Solving Math Quiz: Evaluating the Ability of Large Reasoning Models to Ask for Informationcited by 1
2026Game of Thought: Robust Information Seeking with Large Language Models Using Game Theorycited by 1
2026ALIVE: Awakening LLM Reasoning via Adversarial Learning and Instructive Verbal Evaluationcited by 1
2026Learning to Learn from Language Feedback with Social Meta-Learningcited by 1
2026AutoOR: Scalably Post-training LLMs to Autoformulate Operations Research Problemscited by 1
2026ASPI: Seeking Ambiguity Clarification Amplifies Prompt Injection Vulnerability in LLM Agentscited by 1
2026Interactive Task Alignment as a POMDPcited by 1
2026InsufficiencyBench: Evaluating LLM legal advice on underspecified user queriescited by 1
2025GSM-Agent: Understanding Agentic Reasoning Using Controllable Environmentscited by 0
2026Don't Start What You Can't Finish: A Counterfactual Audit of Support-State Triage in LLM Agentscited by 0
2026Learning When Not to Decide: A Framework for Overcoming Factual Presumptuousness in AI Adjudicationcited by 0
2026When to Ask a Question: Understanding Communication Strategies in Generative AI Toolscited by 0
2026Where Knowledge and Authority Sit Changes What an Agent Benchmark Can Resolvecited by 0
2026MIRA-Math: A Benchmark for Minimal Information Requesting and Mathematical Reasoningcited by 0
2026Automatically Evolving Prompt Guidelines for Task-Specific Optimizationcited by 0
2026FlowEdit: Information-Theoretic Control of LLM Reasoning Flows for Ill-posed Problems Involving Conflictscited by 0
2026From Sports to Safety: Benchmarking Proactive Risk Inference in MLLMscited by 0
2026Ask-E: An Environment for Calibrated Question Generationcited by 0
2026CLAIM: Leading Open-domain Active Clarification of Large Language Models with Uncertainty Measurementcited by 0
2026FinRiskAtlas: Decision-Aligned Evaluation of Large Language Models for Financial Risk Reviewcited by 0
2026PCFBench: A Diagnostic Benchmark for Product Carbon Footprint Estimationcited by 0
2026How You Ask Shapes What You Get: A Theory-Seeded Measurement of Articulation in Advice-Seeking LLM Conversationscited by 0
2026R2A: Learning Persona Policies Through Persona Representation Learning and Runtime Alignmentcited by 0

50 most cited of 63 All 63 (JSON)

Similar, not cited

Close in wording, no citation either way