mlsanityData to 14 Oct 2026

MediQ: Question-Asking LLMs and a Benchmark for Reliable Interactive Clinical Reasoning

arXiv 2406.00922 ↗First posted 3 Jun 2024cs.CL cs.AIcited by 154

On the map: LLM reasoning → Clinical and biomedical applications

Abstract (arXiv)

Users typically engage with LLMs interactively, yet most existing benchmarks evaluate them in a static, single-turn format, posing reliability concerns in interactive scenarios. We identify a key obstacle towards reliability: LLMs are trained to answer any question, even with incomplete context or insufficient knowledge. In this paper, we propose to change the static paradigm to an interactive one, develop systems that proactively ask questions to gather more information and respond reliably, and introduce an benchmark - MediQ - to evaluate question-asking ability in LLMs. MediQ simulates clinical interactions consisting of a Patient System and an adaptive Expert System; with potentially incomplete initial information, the Expert refrains from making diagnostic decisions when unconfident, and instead elicits missing details via follow-up questions. We provide a pipeline to convert single-turn medical benchmarks into an interactive format. Our results show that directly prompting state-of-the-art LLMs to ask questions degrades performance, indicating that adapting LLMs to proactive information-seeking settings is nontrivial. We experiment with abstention strategies to better estimate model confidence and decide when to ask questions, improving diagnostic accuracy by 22.3%; however, performance still lags compared to an (unrealistic in practice) upper bound with complete information upfront. Further analyses show improved interactive performance with filtering irrelevant contexts and reformatting conversations. Overall, we introduce a novel problem towards LLM reliability, an interactive MediQ benchmark and a novel question-asking system, and highlight directions to extend LLMs' information-seeking abilities in critical domains.

Key facts. Cited by 154 papers in the map (data through 14 Oct 2026). Placed in Clinical and biomedical applications, one of 17 directions in LLM reasoning. Builds on 27 of its 47 references that are in the map. First posted on arXiv 3 Jun 2024.

Builds on

27 of 47 references.

References · 27

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  4. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  5. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,998
  6. 2022Large Language Models Encode Clinical Knowledgecited by 1,830
  7. 2020What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Examscited by 1,356
  8. 2019PubMedQA: A Dataset for Biomedical Research Question Answeringcited by 1,288
  9. 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,282
  10. 2023Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formattingcited by 817
  11. 2023Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedbackcited by 812
  12. 2022MedMCQA : A Large-scale Multi-Subject Multi-Choice Dataset for Medical domain Question Answeringcited by 543
15 more
  1. 2023Towards Expert-Level Medical Question Answering with Large Language Modelscited by 384
  2. 2023A Survey of Hallucination in Large Foundation Modelscited by 368
  3. 2023Multi-Agent Collaboration: Harnessing the Power of Intelligent LLM Agentscited by 333
  4. 2018Decoupling Strategy and Generation in Negotiation Dialoguescited by 162
  5. 2023Med-HALT: Medical Domain Hallucination Test for Large Language Modelscited by 157
  6. 2022Deep Bidirectional Language-Knowledge Graph Pretrainingcited by 156
  7. 2023Eliciting Human Preferences with Language Modelscited by 92
  8. 2022DDXPlus: A New Dataset For Automatic Medical Diagnosiscited by 88
  9. 2023PMC-LLaMA: Towards Building Open-source Language Models for Medicinecited by 88
  10. 2021Few-Shot Self-Rationalization with Natural Language Promptscited by 83
  11. 2023Knowledge Card: Filling LLMs' Knowledge Gaps with Plug-in Specialized Language Modelscited by 46
  12. 2024Uncertainty of Thoughts: Uncertainty-Aware Planning Enhances Information Seeking in Large Language Modelscited by 39
  13. 2021Semi-Supervised Variational Reasoning for Medical Dialogue Generationcited by 30
  14. 2024Confidence Calibration and Rationalization for LLMs via Multi-Agent Deliberationcited by 26
  15. 2024Your Co-Workers Matter: Evaluating Collaborative Capabilities of Language Models in Blocks Worldcited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 64; 2026: 88.

154 papers.

Sort:

By area:

  1. 2025AbstentionBench: Reasoning LLMs Fail on Unanswerable Questionscited by 145
  2. 2025Sequential Diagnosis with Language Modelscited by 60
  3. 2026Mind the Sim2Real Gap in User Simulation for Agentic Taskscited by 54
  4. 2025ConfTuner: Training Large Language Models to Express Their Confidence Verballycited by 39
  5. 2025Ambig-SWE: Interactive Agents to Overcome Underspecificity in Software Engineeringcited by 38
  6. 2025QoQ-Med: Building Multimodal Clinical Foundation Models with Domain-Aware GRPO Trainingcited by 38
  7. 2025ChatBench: From Static Benchmarks to Human-AI Evaluationcited by 34
  8. 2025PatientSim: A Persona-Driven Simulator for Realistic Doctor-Patient Interactionscited by 32
  9. 2024HealthQ: Unveiling Questioning Capabilities of LLM Chains in Healthcare Conversationscited by 31
  10. 2025Limitations of Large Language Models in Clinical Problem-Solving Arising from Inflexible Reasoningcited by 31
  11. 2025MEDDxAgent: A Unified Modular Agent Framework for Explainable Automatic Differential Diagnosiscited by 23
  12. 2025MedAgentGym: A Scalable Agentic Training Environment for Code-Centric Reasoning in Biomedical Data Sciencecited by 23
38 more
  1. 2025From Passive to Active Reasoning: Can Large Language Models Ask the Right Questions under Incomplete Information?cited by 23
  2. 20253MDBench: Medical Multimodal Multi-agent Dialogue Benchmarkcited by 16
  3. 2025Cancer-Myth: Evaluating Large Language Models on Patient Questions with False Presuppositionscited by 15
  4. 2025Language Agents for Hypothesis-driven Clinical Decision Making with Reinforcement Learningcited by 15
  5. 2026Agentic Abstention: Do Agents Know When to Stop Instead of Act?cited by 15
  6. 2026Abstain-R1: Calibrated Abstention and Post-Refusal Clarification via Verifiable RLcited by 14
  7. 2025ProMed: Shapley Information Gain Guided Reinforcement Learning for Proactive Medical LLMscited by 13
  8. 2026MedDialogRubrics: A Comprehensive Benchmark and Evaluation Framework for Multi-turn Medical Consultations in Large Language Modelscited by 12
  9. 2025From Scores to Steps: Diagnosing and Improving LLM Performance in Evidence-Based Medical Calculationscited by 11
  10. 2026Over-Searching in Search-Augmented Large Language Modelscited by 11
  11. 2025Follow-up Question Generation For Enhanced Patient-Provider Conversationscited by 10
  12. 2025The Dialogue That Heals: A Comprehensive Evaluation of Doctor Agents' Inquiry Capabilitycited by 10
  13. 2025TALES: A Taxonomy and Analysis of Cultural Representations in LLM-generated Storiescited by 10
  14. 2026When Agents Say One Thing and Do Another: Validating Elicited Beliefs from LLMscited by 10
  15. 2025RefusalBench: Generative Evaluation of Selective Refusal in Grounded Language Modelscited by 9
  16. 2025Exploring the Inquiry-Diagnosis Relationship with Advanced Patient Simulatorscited by 8
  17. 2026EvoClinician: A Self-Evolving Agent for Multi-Turn Medical Diagnosis via Test-Time Evolutionary Learningcited by 8
  18. 2026MT-PingEval: Evaluating Multi-Turn Collaboration with Private Information Gamescited by 8
  19. 2026ATPO: Adaptive Tree Policy Optimization for Multi-Turn Medical Dialoguecited by 8
  20. 2025Conformal Information Pursuit for Interactively Guiding Large Language Modelscited by 7
  21. 2025KnowGuard: Knowledge-Driven Abstention for Multi-Round Clinical Reasoningcited by 7
  22. 2025MedKGI: Iterative Differential Diagnosis with Medical Knowledge Graphs and Information-Guided Inquiringcited by 7
  23. 2024INTERACT: Enabling Interactive, Question-Driven Learning in Large Language Modelscited by 6
  24. 2025Feedback-Aware Monte Carlo Tree Search for Efficient Information Seeking in Goal-Oriented Conversationscited by 6
  25. 2025PPMI: Privacy-Preserving LLM Interaction with Socratic Chain-of-Thought Reasoning and Homomorphically Encrypted Vector Databasescited by 6
  26. 2025Completion neq Collaboration: Scaling Collaborative Effort with Agentscited by 6
  27. 2025MTBBench: A Multimodal Sequential Clinical Decision-Making Benchmark in Oncologycited by 6
  28. 2026MoCo: A One-Stop Shop for Model Collaboration Researchcited by 6
  29. 2025Medical Reasoning in LLMs: An In-Depth Analysis of DeepSeek R1cited by 5
  30. 2025From Promising Capability to Pervasive Bias: Assessing Large Language Models for Emergency Department Triagecited by 5
  31. 2025MedPAIR: Measuring Physicians and AI Relevance Alignment in Medical Question Answeringcited by 5
  32. 2025"What's Up, Doc?": Analyzing How Users Seek Health Information in Large-Scale Conversational AI Datasetscited by 5
  33. 2025Referential ambiguity and clarification requests: comparing human and LLM behaviourcited by 5
  34. 2025QuarkMed Medical Foundation Model Technical Reportcited by 5
  35. 2025Honesty over Accuracy: Trustworthy Language Models through Reinforced Hesitationcited by 5
  36. 2026LLMs are not (consistently) Bayesian: Quantifying internal (in)consistencies of LLMs' probabilistic beliefscited by 5
  37. 2025Recursive Decomposition of Logical Thoughts: Framework for Superior Reasoning and Knowledge Propagation in Large Language Modelscited by 4
  38. 2025Dialogue is Better Than Monologue: Instructing Medical LLMs via Strategical Conversationscited by 4

50 most cited of 154 All 154 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025ALFA: Aligning LLMs to Ask Good Questions A Case Study in Clinical Reasoning
  2. 2025Structured Outputs Enable General-Purpose LLMs to be Medical Experts
  3. 2024LLM-MedQA: Enhancing Medical Question Answering through Case Studies in Large Language Models
  4. 2025MedOmni-45deg: A Safety-Performance Benchmark for Reasoning-Oriented LLMs in Medicine
  5. 2025Building a Human-Verified Clinical Reasoning Dataset via a Human LLM Hybrid Pipeline for Trustworthy Medical AI
  6. 2026Medmarks: A Comprehensive Open-Source LLM Benchmark Suite for Medical Tasks
  7. 2025Automating Expert-Level Medical Reasoning Evaluation of Large Language Models
  8. 2025Med-R2: Crafting Trustworthy LLM Physicians via Retrieval and Reasoning of Evidence-Based Medicine