mlsanityData to 7 Oct 2026

HealthQ: Unveiling Questioning Capabilities of LLM Chains in Healthcare Conversations

arXiv 2409.19487 ↗First posted 28 Sep 2024cs.CL cs.LGcited by 31

Not on the map. Why?

Abstract (arXiv)

Effective patient care in digital healthcare requires large language models (LLMs) that not only answer questions but also actively gather critical information through well-crafted inquiries. This paper introduces HealthQ, a novel framework for evaluating the questioning capabilities of LLM healthcare chains. By implementing advanced LLM chains, including Retrieval-Augmented Generation (RAG), Chain of Thought (CoT), and reflective chains, HealthQ assesses how effectively these chains elicit comprehensive and relevant patient information. To achieve this, we integrate an LLM judge to evaluate generated questions across metrics such as specificity, relevance, and usefulness, while aligning these evaluations with traditional Natural Language Processing (NLP) metrics like ROUGE and Named Entity Recognition (NER)-based set comparisons. We validate HealthQ using two custom datasets constructed from public medical datasets, ChatDoctor and MTS-Dialog, and demonstrate its robustness across multiple LLM judge models, including GPT-3.5, GPT-4, and Claude. Our contributions are threefold: we present the first systematic framework for assessing questioning capabilities in healthcare conversations, establish a model-agnostic evaluation methodology, and provide empirical evidence linking high-quality questions to improved patient information elicitation.

Key facts. Cited by 31 papers in the map (data through 7 Oct 2026). Builds on 31 of its 52 references that are in the map. First posted on arXiv 28 Sep 2024.

Builds on

31 of 52 references.

References · 31

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  5. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  6. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  7. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,800
  8. 2020What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Examscited by 1,324
  9. 2023ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debatecited by 724
  10. 2023Capabilities of GPT-4 on Medical Challenge Problemscited by 630
  11. 2022MedMCQA : A Large-scale Multi-Subject Multi-Choice Dataset for Medical domain Question Answeringcited by 525
  12. 2023Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyondcited by 409
19 more
  1. 2023ChatDoctor: A Medical Chat Model Fine-Tuned on a Large Language Model Meta-AI (LLaMA) Using Medical Domain Knowledgecited by 324
  2. 2022GatorTron: A Large Clinical Language Model to Unlock Patient Information from Unstructured Electronic Health Recordscited by 238
  3. 2024MediQ: Question-Asking LLMs and a Benchmark for Reliable Interactive Clinical Reasoningcited by 145
  4. 2018emrQA: A Large Corpus for Question Answering on Electronic Medical Recordscited by 134
  5. 2023Conversational Health Agents: A Personalized LLM-Powered Agent Frameworkcited by 91
  6. 2018Medical Exam Question Answering with Large-scale Reading Comprehensioncited by 29
  7. 2024ALCM: Autonomous LLM-Augmented Causal Discovery Frameworkcited by 29
  8. 2024A Decision-Language Model (DLM) for Dynamic Restless Multi-Armed Bandit Tasks in Public Healthcited by 24
  9. 2024ChatDiet: Empowering Personalized Nutrition-Oriented Food Recommender Chatbots through an LLM-Augmented Frameworkcited by 20
  10. 2024Differential Private Federated Transfer Learning for Mental Health Monitoring in Everyday Settings: A Case Study on Stress Detectioncited by 16
  11. 2023Towards a Pretrained Model for Restless Bandits via Multi-arm Generalizationcited by 9
  12. 2023Scalable Neural Network Kernelscited by 8
  13. 2024Integrating Wearable Sensor Data and Self-reported Diaries for Personalized Affect Forecastingcited by 7
  14. 2024Enhancing Performance and User Engagement in Everyday Stress Monitoring: A Context-Aware Active Reinforcement Learning Approachcited by 7
  15. 2024ECG Unveiled: Analysis of Client Re-identification Risks in Real-World ECG Datasetscited by 7
  16. 2024Efflex: Efficient and Flexible Pipeline for Spatio-Temporal Trajectory Graph Modeling and Representation Learningcited by 5
  17. 2024The Bandit Whisperer: Communication Learning for Restless Banditscited by 5
  18. 2024Improving the Prediction of Individual Engagement in Recommendations Using Cognitive Modelscited by 3
  19. 2024Loneliness Forecasting Using Multi-modal Wearable and Mobile Sensing in Everyday Settingscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 5; 2025: 18; 2026: 8.

31 papers.

Sort:

By area:

  1. 2024From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judgecited by 480
  2. 2025Multimodal Chain-of-Thought Reasoning: A Comprehensive Surveycited by 176
  3. 2024Feedback-Guided Extraction of Knowledge Base from Retrieval-Augmented LLM Applicationscited by 36
  4. 2025ALFA: Aligning LLMs to Ask Good Questions A Case Study in Clinical Reasoningcited by 24
  5. 2024Pirates of the RAG: Adaptively Attacking LLMs to Leak Knowledge Basescited by 17
  6. 2025Learning Musical Representations for Music Performance Question Answeringcited by 17
  7. 2025ProMed: Shapley Information Gain Guided Reinforcement Learning for Proactive Medical LLMscited by 12
  8. 2025Privacy-Aware RAG: Secure and Isolated Knowledge Retrievalcited by 10
  9. 2024Towards Foundation-model-based Multiagent System to Accelerate AI for Social Impactcited by 9
  10. 2025Follow-up Question Generation For Enhanced Patient-Provider Conversationscited by 9
  11. 2025Large Language Models for Security Operations Centers: A Comprehensive Surveycited by 9
  12. 2025MedCoT-RAG: Causal Chain-of-Thought RAG for Medical Question Answeringcited by 5
19 more
  1. 2025KnowGuard: Knowledge-Driven Abstention for Multi-Round Clinical Reasoningcited by 5
  2. 2025Linkage Attacks Expose Identity Risks in Public ECG Data Sharingcited by 4
  3. 2025TransECG: Leveraging Transformers for Explainable ECG Re-identification Risk Analysiscited by 3
  4. 2026InfoGatherer: Principled Information Seeking via Evidence Retrieval and Strategic Questioningcited by 3
  5. 2026CARE-ECG: Causal Agent-based Reasoning for Explainable and Counterfactual ECG Interpretationcited by 2
  6. 2025Toward Revealing Nuanced Biases in Medical LLMscited by 1
  7. 2025HealthBranches: Synthesizing Clinically-Grounded Question Answering Datasets via Decision Pathwayscited by 1
  8. 2025Diagnosing Hallucination Risk in AI Surgical Decision-Support: A Sequential Framework for Sequential Validationcited by 1
  9. 2026SoK: Privacy-aware LLM in Healthcare: Threat Model, Privacy Techniques, Challenges and Recommendationscited by 1
  10. 2024AI Learning Algorithms: Deep Learning, Hybrid Models, and Large-Scale Model Integrationcited by 0
  11. 2025HIVMedQA: Benchmarking large language models for HIV medical decision supportcited by 0
  12. 2025Provably Secure Retrieval-Augmented Generationcited by 0
  13. 2025When Explainability Meets Privacy: An Investigation at the Intersection of Post-hoc Explainability and Differential Privacy in the Context of Natural Language Processingcited by 0
  14. 2025Real-Time Health Analytics Using Ontology-Driven Complex Event Processing and LLM Reasoning: A Tuberculosis Case Studycited by 0
  15. 2026Dialogue to Question Generation for Evidence-based Medical Guideline Agent Developmentcited by 0
  16. 2026Membership Inference Attacks Expose Participation Privacy in ECG Foundation Encoderscited by 0
  17. 2026RPS: Information Elicitation with Reinforcement Prompt Selectioncited by 0
  18. 2026PerCaM-Health: Personalized Dynamic Causal Graphs for Healthcare Reasoningcited by 0
  19. 2026Beyond Correctness: Validity-Oriented Evaluation of Biomedical LLM Judgescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Tool Calling: Enhancing Medication Consultation via Retrieval-Augmented Large Language Models
  2. 2025Conversation AI Dialog for Medicare powered by Finetuning and Retrieval Augmented Generation
  3. 2025RephQA: Evaluating Readability of Large Language Models in Public Health Question Answering
  4. 2024The Geometry of Queries: Query-Based Innovations in Retrieval-Augmented Generation for Healthcare QA
  5. 2026Healthier LLMs: Retrieval-Augmented Generation for Public Health Question Answering
  6. 2025Building a Human-Verified Clinical Reasoning Dataset via a Human LLM Hybrid Pipeline for Trustworthy Medical AI
  7. 2025Med-R2: Crafting Trustworthy LLM Physicians via Retrieval and Reasoning of Evidence-Based Medicine
  8. 2023MKRAG: Medical Knowledge Retrieval Augmented Generation for Medical Question Answering