# HealthQ: Unveiling Questioning Capabilities of LLM Chains in Healthcare Conversations

arXiv: https://arxiv.org/abs/2409.19487 · HTML: https://mlsanity.com/paper/2409.19487/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 52 references that are in the map.
- First posted on arXiv 28 Sep 2024.

## Builds on (most cited first, 30 of 31)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection](https://mlsanity.com/paper/2310.11511/) (2023)
- [What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Exams](https://mlsanity.com/paper/2009.13081/) (2020)
- [ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debate](https://mlsanity.com/paper/2308.07201/) (2023)
- [Capabilities of GPT-4 on Medical Challenge Problems](https://mlsanity.com/paper/2303.13375/) (2023)
- [MedMCQA : A Large-scale Multi-Subject Multi-Choice Dataset for Medical domain Question Answering](https://mlsanity.com/paper/2203.14371/) (2022)
- [Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyond](https://mlsanity.com/paper/2304.13712/) (2023)
- [ChatDoctor: A Medical Chat Model Fine-Tuned on a Large Language Model Meta-AI (LLaMA) Using Medical Domain Knowledge](https://mlsanity.com/paper/2303.14070/) (2023)
- [GatorTron: A Large Clinical Language Model to Unlock Patient Information from Unstructured Electronic Health Records](https://mlsanity.com/paper/2203.03540/) (2022)
- [MediQ: Question-Asking LLMs and a Benchmark for Reliable Interactive Clinical Reasoning](https://mlsanity.com/paper/2406.00922/) (2024)
- [emrQA: A Large Corpus for Question Answering on Electronic Medical Records](https://mlsanity.com/paper/1809.00732/) (2018)
- [Conversational Health Agents: A Personalized LLM-Powered Agent Framework](https://mlsanity.com/paper/2310.02374/) (2023)
- [Medical Exam Question Answering with Large-scale Reading Comprehension](https://mlsanity.com/paper/1802.10279/) (2018)
- [ALCM: Autonomous LLM-Augmented Causal Discovery Framework](https://mlsanity.com/paper/2405.01744/) (2024)
- [A Decision-Language Model (DLM) for Dynamic Restless Multi-Armed Bandit Tasks in Public Health](https://mlsanity.com/paper/2402.14807/) (2024)
- [ChatDiet: Empowering Personalized Nutrition-Oriented Food Recommender Chatbots through an LLM-Augmented Framework](https://mlsanity.com/paper/2403.00781/) (2024)
- [Differential Private Federated Transfer Learning for Mental Health Monitoring in Everyday Settings: A Case Study on Stress Detection](https://mlsanity.com/paper/2402.10862/) (2024)
- [Scalable Neural Network Kernels](https://mlsanity.com/paper/2310.13225/) (2023)
- [Towards a Pretrained Model for Restless Bandits via Multi-arm Generalization](https://mlsanity.com/paper/2310.14526/) (2023)
- [Integrating Wearable Sensor Data and Self-reported Diaries for Personalized Affect Forecasting](https://mlsanity.com/paper/2403.13841/) (2024)
- [Enhancing Performance and User Engagement in Everyday Stress Monitoring: A Context-Aware Active Reinforcement Learning Approach](https://mlsanity.com/paper/2407.08215/) (2024)
- [ECG Unveiled: Analysis of Client Re-identification Risks in Real-World ECG Datasets](https://mlsanity.com/paper/2408.10228/) (2024)
- [Efflex: Efficient and Flexible Pipeline for Spatio-Temporal Trajectory Graph Modeling and Representation Learning](https://mlsanity.com/paper/2404.12400/) (2024)
- [The Bandit Whisperer: Communication Learning for Restless Bandits](https://mlsanity.com/paper/2408.05686/) (2024)
- [Improving the Prediction of Individual Engagement in Recommendations Using Cognitive Models](https://mlsanity.com/paper/2408.16147/) (2024)

## Built on by (most cited first, 30 of 31)

- [From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judge](https://mlsanity.com/paper/2411.16594/) (2024)
- [Multimodal Chain-of-Thought Reasoning: A Comprehensive Survey](https://mlsanity.com/paper/2503.12605/) (2025)
- [Feedback-Guided Extraction of Knowledge Base from Retrieval-Augmented LLM Applications](https://mlsanity.com/paper/2411.14110/) (2024)
- [ALFA: Aligning LLMs to Ask Good Questions A Case Study in Clinical Reasoning](https://mlsanity.com/paper/2502.14860/) (2025)
- [Pirates of the RAG: Adaptively Attacking LLMs to Leak Knowledge Bases](https://mlsanity.com/paper/2412.18295/) (2024)
- [Learning Musical Representations for Music Performance Question Answering](https://mlsanity.com/paper/2502.06710/) (2025)
- [ProMed: Shapley Information Gain Guided Reinforcement Learning for Proactive Medical LLMs](https://mlsanity.com/paper/2508.13514/) (2025)
- [Privacy-Aware RAG: Secure and Isolated Knowledge Retrieval](https://mlsanity.com/paper/2503.15548/) (2025)
- [Follow-up Question Generation For Enhanced Patient-Provider Conversations](https://mlsanity.com/paper/2503.17509/) (2025)
- [Towards Foundation-model-based Multiagent System to Accelerate AI for Social Impact](https://mlsanity.com/paper/2412.07880/) (2024)
- [Large Language Models for Security Operations Centers: A Comprehensive Survey](https://mlsanity.com/paper/2509.10858/) (2025)
- [KnowGuard: Knowledge-Driven Abstention for Multi-Round Clinical Reasoning](https://mlsanity.com/paper/2509.24816/) (2025)
- [MedCoT-RAG: Causal Chain-of-Thought RAG for Medical Question Answering](https://mlsanity.com/paper/2508.15849/) (2025)
- [Linkage Attacks Expose Identity Risks in Public ECG Data Sharing](https://mlsanity.com/paper/2508.15850/) (2025)
- [TransECG: Leveraging Transformers for Explainable ECG Re-identification Risk Analysis](https://mlsanity.com/paper/2503.13495/) (2025)
- [SoK: Privacy-aware LLM in Healthcare: Threat Model, Privacy Techniques, Challenges and Recommendations](https://mlsanity.com/paper/2601.10004/) (2026)
- [InfoGatherer: Principled Information Seeking via Evidence Retrieval and Strategic Questioning](https://mlsanity.com/paper/2603.05909/) (2026)
- [CARE-ECG: Causal Agent-based Reasoning for Explainable and Counterfactual ECG Interpretation](https://mlsanity.com/paper/2604.10420/) (2026)
- [Toward Revealing Nuanced Biases in Medical LLMs](https://mlsanity.com/paper/2507.21176/) (2025)
- [HealthBranches: Synthesizing Clinically-Grounded Question Answering Datasets via Decision Pathways](https://mlsanity.com/paper/2508.07308/) (2025)
- [Diagnosing Hallucination Risk in AI Surgical Decision-Support: A Sequential Framework for Sequential Validation](https://mlsanity.com/paper/2511.00588/) (2025)
- [AI Learning Algorithms: Deep Learning, Hybrid Models, and Large-Scale Model Integration](https://mlsanity.com/paper/2410.09186/) (2024)
- [HIVMedQA: Benchmarking large language models for HIV medical decision support](https://mlsanity.com/paper/2507.18143/) (2025)
- [Provably Secure Retrieval-Augmented Generation](https://mlsanity.com/paper/2508.01084/) (2025)
- [When Explainability Meets Privacy: An Investigation at the Intersection of Post-hoc Explainability and Differential Privacy in the Context of Natural Language Processing](https://mlsanity.com/paper/2508.10482/) (2025)
- [Real-Time Health Analytics Using Ontology-Driven Complex Event Processing and LLM Reasoning: A Tuberculosis Case Study](https://mlsanity.com/paper/2510.09646/) (2025)
- [Dialogue to Question Generation for Evidence-based Medical Guideline Agent Development](https://mlsanity.com/paper/2603.23937/) (2026)
- [Membership Inference Attacks Expose Participation Privacy in ECG Foundation Encoders](https://mlsanity.com/paper/2604.10424/) (2026)
- [RPS: Information Elicitation with Reinforcement Prompt Selection](https://mlsanity.com/paper/2604.13817/) (2026)
- [PerCaM-Health: Personalized Dynamic Causal Graphs for Healthcare Reasoning](https://mlsanity.com/paper/2605.07267/) (2026)

## Abstract (arXiv)

Effective patient care in digital healthcare requires large language models (LLMs) that not only answer questions but also actively gather critical information through well-crafted inquiries. This paper introduces HealthQ, a novel framework for evaluating the questioning capabilities of LLM healthcare chains. By implementing advanced LLM chains, including Retrieval-Augmented Generation (RAG), Chain of Thought (CoT), and reflective chains, HealthQ assesses how effectively these chains elicit comprehensive and relevant patient information. To achieve this, we integrate an LLM judge to evaluate generated questions across metrics such as specificity, relevance, and usefulness, while aligning these evaluations with traditional Natural Language Processing (NLP) metrics like ROUGE and Named Entity Recognition (NER)-based set comparisons. We validate HealthQ using two custom datasets constructed from public medical datasets, ChatDoctor and MTS-Dialog, and demonstrate its robustness across multiple LLM judge models, including GPT-3.5, GPT-4, and Claude. Our contributions are threefold: we present the first systematic framework for assessing questioning capabilities in healthcare conversations, establish a model-agnostic evaluation methodology, and provide empirical evidence linking high-quality questions to improved patient information elicitation.