# InfoGatherer: Principled Information Seeking via Evidence Retrieval and Strategic Questioning

arXiv: https://arxiv.org/abs/2603.05909 · HTML: https://mlsanity.com/paper/2603.05909/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 71 references that are in the map.
- First posted on arXiv 6 Mar 2026.

## Builds on (most cited first, 30 of 34)

- [What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Exams](https://mlsanity.com/paper/2009.13081/) (2020)
- [Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback](https://mlsanity.com/paper/2305.14975/) (2023)
- [AmbigQA: Answering Ambiguous Open-domain Questions](https://mlsanity.com/paper/2004.10645/) (2020)
- [Challenges and Strategies in Cross-Cultural NLP](https://mlsanity.com/paper/2203.10020/) (2022)
- [WebDancer: Towards Autonomous Information Seeking Agency](https://mlsanity.com/paper/2505.22648/) (2025)
- [Learning to Ask: Neural Question Generation for Reading Comprehension](https://mlsanity.com/paper/1705.00106/) (2017)
- [MediQ: Question-Asking LLMs and a Benchmark for Reliable Interactive Clinical Reasoning](https://mlsanity.com/paper/2406.00922/) (2024)
- [Asking Clarifying Questions in Open-Domain Information-Seeking Conversations](https://mlsanity.com/paper/1907.06554/) (2019)
- [Large Language Models in Law: A Survey](https://mlsanity.com/paper/2312.03718/) (2023)
- [Benchmarking LLMs via Uncertainty Quantification](https://mlsanity.com/paper/2401.12794/) (2024)
- [STaR-GATE: Teaching Language Models to Ask Clarifying Questions](https://mlsanity.com/paper/2403.19154/) (2024)
- [Eliciting Human Preferences with Language Models](https://mlsanity.com/paper/2310.11589/) (2023)
- [Modeling Future Conversation Turns to Teach LLMs to Ask Clarifying Questions](https://mlsanity.com/paper/2410.13788/) (2024)
- [SimpleDeepSearcher: Deep Information Seeking via Web-Powered Reasoning Trajectory Synthesis](https://mlsanity.com/paper/2505.16834/) (2025)
- [Answer-based Adversarial Training for Generating Clarification Questions](https://mlsanity.com/paper/1904.02281/) (2019)
- [Learning to Ask: When LLM Agents Meet Unclear Instruction](https://mlsanity.com/paper/2409.00557/) (2024)
- [Ask what's missing and what's useful: Improving Clarification Question Generation using Global Knowledge](https://mlsanity.com/paper/2104.06828/) (2021)
- [A Reasoning-Focused Legal Retrieval Benchmark](https://mlsanity.com/paper/2505.03970/) (2025)
- [HealthQ: Unveiling Questioning Capabilities of LLM Chains in Healthcare Conversations](https://mlsanity.com/paper/2409.19487/) (2024)
- [A Comprehensive Evaluation of Large Language Models on Legal Judgment Prediction](https://mlsanity.com/paper/2310.11761/) (2023)
- [INSCIT: Information-Seeking Conversations with Mixed-Initiative Interactions](https://mlsanity.com/paper/2207.00746/) (2022)
- [ALFA: Aligning LLMs to Ask Good Questions A Case Study in Clinical Reasoning](https://mlsanity.com/paper/2502.14860/) (2025)
- [Large Language Models Perform Diagnostic Reasoning](https://mlsanity.com/paper/2307.08922/) (2023)
- [LLMs Don't Know Their Own Decision Boundaries: The Unreliability of Self-Generated Counterfactual Explanations](https://mlsanity.com/paper/2509.09396/) (2025)
- [Teaching Language Models To Gather Information Proactively](https://mlsanity.com/paper/2507.21389/) (2025)
- [FOLLOWUPQG: Towards Information-Seeking Follow-up Question Generation](https://mlsanity.com/paper/2309.05007/) (2023)
- [Your Co-Workers Matter: Evaluating Collaborative Capabilities of Language Models in Blocks World](https://mlsanity.com/paper/2404.00246/) (2024)
- [QUITE: Quantifying Uncertainty in Natural Language Text in Bayesian Reasoning Scenarios](https://mlsanity.com/paper/2410.10449/) (2024)
- [Intelligent Legal Assistant: An Interactive Clarification System for Legal Question Answering](https://mlsanity.com/paper/2502.07904/) (2025)
- [Is It Bad to Work All the Time? Cross-Cultural Evaluation of Social Norm Biases in GPT-4](https://mlsanity.com/paper/2505.18322/) (2025)

## Built on by (most cited first, 3 of 3)

- [Legal LLM Hallucination Should Be Evaluated as Failure of Legal Warrant](https://mlsanity.com/paper/2609.17546/) (2026)
- [Why Didn't It Check? Unsupported Final Claims and Their Repair in Two Tool-Equipped Language Models](https://mlsanity.com/paper/2608.27768/) (2026)
- [Realize What Matters: Principled Context Representation for Large-Scale Reasoning](https://mlsanity.com/paper/2609.27173/) (2026)

## Abstract (arXiv)

LLMs are increasingly deployed in high-stakes domains such as medical triage and legal assistance, often as document-grounded QA systems in which a user provides a description, relevant sources are retrieved, and an LLM generates a prediction. In practice, initial user queries are often underspecified, and a single retrieval pass is insufficient for reliable decision-making, leading to incorrect and overly confident answers. While follow-up questioning can elicit missing information, existing methods typically depend on implicit, unstructured confidence signals from the LLM, making it difficult to determine what remains unknown, what information matters most, and when to stop asking questions. We propose InfoGatherer, a framework that gathers missing information from two complementary sources: retrieved domain documents and targeted follow-up questions to the user. InfoGatherer models uncertainty using Dempster-Shafer belief assignments over a structured evidential network, enabling principled fusion of incomplete and potentially contradictory evidence from both sources without prematurely collapsing to a definitive answer. Across legal and medical tasks, InfoGatherer outperforms strong baselines while requiring fewer turns. By grounding uncertainty in formal evidential theory rather than heuristic LLM signals, InfoGatherer moves towards trustworthy, interpretable decision support in domains where reliability is critical.