mlsanityData to 14 Oct 2026

Foundation Metrics for Evaluating Effectiveness of Healthcare Conversations Powered by Generative AI

arXiv 2309.12444 ↗First posted 21 Sep 2023cs.CLcited by 32

Not on the map. Why?

Abstract (arXiv)

Generative Artificial Intelligence is set to revolutionize healthcare delivery by transforming traditional patient care into a more personalized, efficient, and proactive process. Chatbots, serving as interactive conversational models, will probably drive this patient-centered transformation in healthcare. Through the provision of various services, including diagnosis, personalized lifestyle recommendations, and mental health support, the objective is to substantially augment patient health outcomes, all the while mitigating the workload burden on healthcare providers. The life-critical nature of healthcare applications necessitates establishing a unified and comprehensive set of evaluation metrics for conversational models. Existing evaluation metrics proposed for various generic large language models (LLMs) demonstrate a lack of comprehension regarding medical and health concepts and their significance in promoting patients' well-being. Moreover, these metrics neglect pivotal user-centered aspects, including trust-building, ethics, personalization, empathy, user comprehension, and emotional support. The purpose of this paper is to explore state-of-the-art LLM-based evaluation metrics that are specifically applicable to the assessment of interactive conversational models in healthcare. Subsequently, we present an comprehensive set of evaluation metrics designed to thoroughly assess the performance of healthcare chatbots from an end-user perspective. These metrics encompass an evaluation of language processing abilities, impact on real-world clinical tasks, and effectiveness in user-interactive conversations. Finally, we engage in a discussion concerning the challenges associated with defining and implementing these metrics, with particular emphasis on confounding factors such as the target audience, evaluation methods, and prompt techniques involved in the evaluation process.

Key facts. Cited by 32 papers in the map (data through 14 Oct 2026). Builds on 97 of its 151 references that are in the map. First posted on arXiv 21 Sep 2023.

Builds on

97 of 151 references.

References · 97

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  7. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  8. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  9. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  10. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  11. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
  12. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
38 more
  1. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  2. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  3. 2023A Survey of Large Language Modelscited by 2,576
  4. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  5. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
  6. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,234
  7. 2020Extracting Training Data from Large Language Modelscited by 2,189
  8. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
  9. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  10. 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,947
  11. 2019Language Models as Knowledge Bases?cited by 1,936
  12. 2022Large Language Models Encode Clinical Knowledgecited by 1,830
  13. 2019SuperGlue: Learning Feature Matching with Graph Neural Networkscited by 1,443
  14. 2020What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Examscited by 1,356
  15. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,317
  16. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,255
  17. 2023A Survey on Evaluation of Large Language Modelscited by 1,242
  18. 2022LaMDA: Language Models for Dialog Applicationscited by 1,220
  19. 2023Large Language Models are not Fair Evaluatorscited by 1,085
  20. 2015Teaching Machines to Read and Comprehendcited by 1,049
  21. 2022Large Language Models Are Human-Level Prompt Engineerscited by 1,045
  22. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 992
  23. 2021Measuring Coding Challenge Competence With APPScited by 902
  24. 2023A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivitycited by 894
  25. 2019DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generationcited by 842
  26. 2016Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyondcited by 812
  27. 2023SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Modelscited by 810
  28. 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 734
  29. 2020CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Modelscited by 710
  30. 2021BBQ: A Hand-Built Bias Benchmark for Question Answeringcited by 701
  31. 2017The NarrativeQA Reading Comprehension Challengecited by 597
  32. 2019BLiMP: The Benchmark of Linguistic Minimal Pairs for Englishcited by 564
  33. 2022BioGPT: Generative Pre-trained Transformer for Biomedical Text Generation and Miningcited by 538
  34. 2022Improving alignment of dialogue agents via targeted human judgementscited by 531
  35. 2023DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Modelscited by 518
  36. 2020Towards a Human-like Open-Domain Chatbotcited by 470
  37. 2019MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distancecited by 418
  38. 2021BOLD: Dataset and Metrics for Measuring Biases in Open-Ended Language Generationcited by 412

50 most cited of 97 All 97 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 12; 2025: 14; 2026: 5.

32 papers.

Sort:

By area:

  1. 2023Conversational Health Agents: A Personalized LLM-Powered Agent Frameworkcited by 92
  2. 2024Towards a Holistic Framework for Multimodal Large Language Models in Three-dimensional Brain CT Report Generationcited by 31
  3. 2025The Scales of Justitia: A Comprehensive Survey on Safety Evaluation of LLMscited by 26
  4. 2025Automating Expert-Level Medical Reasoning Evaluation of Large Language Modelscited by 22
  5. 2026HEART: A Unified Benchmark for Assessing Humans and LLMs in Emotional Support Dialoguecited by 13
  6. 2024WorldMedQA-V: a multilingual, multimodal medical examination dataset for multimodal language models evaluationcited by 11
  7. 2025A Scalable Framework for Evaluating Health Language Modelscited by 11
  8. 2024A Review on Generative AI Models for Synthetic Medical Text, Time Series, and Longitudinal Datacited by 10
  9. 2024Building Trust in Mental Health Chatbots: Safety Metrics and LLM-Based Evaluation Toolscited by 9
  10. 2025Uncertainty-Aware Large Language Models for Explainable Disease Diagnosiscited by 9
  11. 2024FEDMEKI: A Benchmark for Scaling Medical Foundation Models via Federated Knowledge Injectioncited by 8
  12. 2024Clinical Insights: A Comprehensive Review of Language Models in Medicinecited by 6
20 more
  1. 2025Applications of Large Models in Medicinecited by 6
  2. 2024Emerging Opportunities of Using Large Language Models for Translation Between Drug Molecules and Indicationscited by 5
  3. 2024FEDKIM: Adaptive Federated Knowledge Injection into Medical Foundation Modelscited by 5
  4. 2024Electrocardiogram-Language Model for Few-Shot Question Answering with Meta Learningcited by 5
  5. 2024Interpretable Differential Diagnosis with Dual-Inference Large Language Modelscited by 4
  6. 2025A Principle-based Framework for the Development and Evaluation of Large Language Models for Health and Wellnesscited by 3
  7. 2024Accuracy and Consistency of LLMs in the Registered Dietitian Exam: The Impact of Prompt Engineering and Knowledge Retrievalcited by 2
  8. 2024Generative AI in Medicinecited by 1
  9. 2025Large Language Models for Cancer Communication: Evaluating Linguistic Quality, Safety, and Accessibility in Generative AIcited by 1
  10. 2025AI-Augmented LLMs Achieve Therapist-Level Responses in Motivational Interviewingcited by 1
  11. 2025Filling in the Clinical Gaps in Benchmark: Case for HealthBench for the Japanese medical systemcited by 1
  12. 2025Personal Care Utility (PCU): Building the Health Infrastructure for Everyday Insight and Guidancecited by 1
  13. 2025Plausibility as Failure: How LLMs and Humans Co-Construct Epistemic Errorcited by 1
  14. 2026Toward Trustworthy Large Language Model Agents in Healthcarecited by 1
  15. 2025Evaluating Large Language Models on the Spanish Medical Intern Resident (MIR) Examination 2024/2025:A Comparative Analysis of Clinical Reasoning and Knowledge Applicationcited by 0
  16. 2025Evaluating Medical LLMs by Levels of Autonomy: A Survey Moving from Benchmarks to Applicationscited by 0
  17. 2025Towards Better Health Conversations: The Benefits of Context-seekingcited by 0
  18. 2026SpheriCity: Designing Trustworthy Conversational AI for Sustainability Decision Supportcited by 0
  19. 2026MindTailor: Personalized Emotional Support via Post History-Grounded Case Formulation and Collaborative Refinementcited by 0
  20. 2026CardioBench: A Real-World Data Benchmark for Evaluating Large Language Models in Clinically Authentic Cardiovascular Care Scenarioscited by 0