# From Passive to Active Reasoning: Can Large Language Models Ask the Right Questions under Incomplete Information?

arXiv: https://arxiv.org/abs/2506.08295 · HTML: https://mlsanity.com/paper/2506.08295/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning.
- Builds on 97 of its 109 references that are in the map.
- First posted on arXiv 9 Jun 2025.

## Placed in

- LLM reasoning / [Reasoning benchmarks and evaluation](https://mlsanity.com/lr/reasoning-benchmarks-and-evaluation/)

## Builds on (most cited first, 30 of 97)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [Reflexion: Language Agents with Verbal Reinforcement Learning](https://mlsanity.com/paper/2303.11366/) (2023)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2004.04906/) (2020)
- [Let's Verify Step by Step](https://mlsanity.com/paper/2305.20050/) (2023)
- [HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering](https://mlsanity.com/paper/1809.09600/) (2018)
- [Program Synthesis with Large Language Models](https://mlsanity.com/paper/2108.07732/) (2021)
- [Self-Refine: Iterative Refinement with Self-Feedback](https://mlsanity.com/paper/2303.17651/) (2023)
- [QLoRA: Efficient Finetuning of Quantized LLMs](https://mlsanity.com/paper/2305.14314/) (2023)
- [Universal and Transferable Adversarial Attacks on Aligned Language Models](https://mlsanity.com/paper/2307.15043/) (2023)
- [TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension](https://mlsanity.com/paper/1705.03551/) (2017)
- [DAPO: An Open-Source LLM Reinforcement Learning System at Scale](https://mlsanity.com/paper/2503.14476/) (2025)
- [REALM: Retrieval-Augmented Language Model Pre-Training](https://mlsanity.com/paper/2002.08909/) (2020)
- [CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge](https://mlsanity.com/paper/1811.00937/) (2018)
- [Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning](https://mlsanity.com/paper/2503.09516/) (2025)
- [Least-to-Most Prompting Enables Complex Reasoning in Large Language Models](https://mlsanity.com/paper/2205.10625/) (2022)

## Built on by (most cited first, 23 of 23)

- [LLM Unlearning with LLM Beliefs](https://mlsanity.com/paper/2510.19422/) (2025)
- [Uncertainty-Aware Clarification in LLM Agents with Information Gain](https://mlsanity.com/paper/2606.03135/) (2026)
- [Reasoned Safety Alignment: Ensuring Jailbreak Defense via Answer-Then-Check](https://mlsanity.com/paper/2509.11629/) (2025)
- [Reference-guided Policy Optimization for Molecular Optimization via LLM Reasoning](https://mlsanity.com/paper/2603.05900/) (2026)
- [On Information Self-Locking in Reinforcement Learning for Active Reasoning of LLM agents](https://mlsanity.com/paper/2603.12109/) (2026)
- [Reducing Belief Deviation in Reinforcement Learning for Active Reasoning](https://mlsanity.com/paper/2510.12264/) (2025)
- [Towards Understanding Valuable Preference Data for Large Language Model Alignment](https://mlsanity.com/paper/2510.13212/) (2025)
- [Grounded in Reality: Learning and Deploying Proactive LLM from Offline Logs](https://mlsanity.com/paper/2510.25441/) (2025)
- [IntentRL: Training Proactive User-intent Agents for Open-ended Deep Research via Reinforcement Learning](https://mlsanity.com/paper/2602.03468/) (2026)
- [The Easy, the Hard, and the Learnable: Confidence and Difficulty-Adaptive Policy Optimization for LLM Reasoning](https://mlsanity.com/paper/2606.07950/) (2026)
- [Benchmarking In-context Experiential Learning Through Repeated Product Recommendations](https://mlsanity.com/paper/2511.22130/) (2025)
- [TeamTR: Trust-Region Fine-Tuning for Multi-Agent LLM Coordination](https://mlsanity.com/paper/2605.15207/) (2026)
- [GSM-Agent: Understanding Agentic Reasoning Using Controllable Environments](https://mlsanity.com/paper/2509.21998/) (2025)
- [When Seeing Is not Enough: Revealing the Limits of Active Reasoning in MLLMs](https://mlsanity.com/paper/2510.15421/) (2025)
- [Automatic Pruning Discovery for Large Language Models](https://mlsanity.com/paper/2511.15390/) (2025)
- [CA-BED: Conversation-Aware Bayesian Experimental Design](https://mlsanity.com/paper/2606.01182/) (2026)
- [DICE: Entropy-Regularized Equilibrium Selection for Stable Multi-Agent LLM Coordination](https://mlsanity.com/paper/2606.08068/) (2026)
- [IsoSci: A Benchmark of Isomorphic Cross-Domain Science Problems for Evaluating Reasoning versus Knowledge Retrieval in LLMs](https://mlsanity.com/paper/2607.01431/) (2026)
- [Automatically Evolving Prompt Guidelines for Task-Specific Optimization](https://mlsanity.com/paper/2607.14105/) (2026)
- [CLAIM: Leading Open-domain Active Clarification of Large Language Models with Uncertainty Measurement](https://mlsanity.com/paper/2608.11631/) (2026)
- [FinRiskAtlas: Decision-Aligned Evaluation of Large Language Models for Financial Risk Review](https://mlsanity.com/paper/2608.25325/) (2026)
- [Verifiable Hidden Dynamics Play: Generating Agentic RL Environments from Solved Mechanisms](https://mlsanity.com/paper/2609.27321/) (2026)
- [How Well Do LLMs Reason with Noisy Evidence? An Active Visual Reasoning Benchmark](https://mlsanity.com/paper/2610.07751/) (2026)

## Abstract (arXiv)

While existing benchmarks probe the reasoning abilities of large language models (LLMs) across diverse domains, they predominantly assess passive reasoning, providing models with all the information needed to reach a solution. By contrast, active reasoning-where an LLM must interact with external systems to acquire missing evidence or data-has received little systematic attention. To address this shortfall, we present AR-Bench, a novel benchmark designed explicitly to evaluate an LLM's active reasoning skills. AR-Bench comprises three task families-detective cases, situation puzzles, and guessing numbers-that together simulate real-world, agentic scenarios and measure performance across commonsense, logical, and symbolic reasoning challenges. Empirical evaluation on AR-Bench demonstrates that contemporary LLMs exhibit pronounced difficulties with active reasoning: they frequently fail to acquire or leverage the information needed to solve tasks. This gap highlights a stark divergence between their passive and active reasoning abilities. Moreover, ablation studies indicate that even advanced strategies, such as tree-based searching or post-training approaches, yield only modest gains and fall short of the levels required for real-world deployment. Collectively, these findings highlight the critical need to advance methodology for active reasoning, e.g., incorporating interactive learning, real-time feedback loops, and environment-aware objectives for training. The benchmark is publicly available at: https://github.com/tmlr-group/AR-Bench.