mlsanityData to 7 Oct 2026

Evaluation of ChatGPT-Generated Medical Responses: A Systematic Review and Meta-Analysis

arXiv 2310.08410 ↗First posted 12 Oct 2023stat.ME stat.MLcited by 13

Not on the map. Why?

Abstract (arXiv)

Large language models such as ChatGPT are increasingly explored in medical domains. However, the absence of standard guidelines for performance evaluation has led to methodological inconsistencies. This study aims to summarize the available evidence on evaluating ChatGPT's performance in medicine and provide direction for future research. We searched ten medical literature databases on June 15, 2023, using the keyword "ChatGPT". A total of 3520 articles were identified, of which 60 were reviewed and summarized in this paper and 17 were included in the meta-analysis. The analysis showed that ChatGPT displayed an overall integrated accuracy of 56% (95% CI: 51%-60%, I2 = 87%) in addressing medical queries. However, the studies varied in question resource, question-asking process, and evaluation metrics. Moreover, many studies failed to report methodological details, including the version of ChatGPT and whether each question was used independently or repeatedly. Our findings revealed that although ChatGPT demonstrated considerable potential for application in healthcare, the heterogeneity of the studies and insufficient reporting may affect the reliability of these results. Further well-designed studies with comprehensive and transparent reporting are needed to evaluate ChatGPT's performance in medicine.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Builds on 3 of its 80 references that are in the map. First posted on arXiv 12 Oct 2023.

Builds on

3 of 80 references.

References · 3

  1. 2022Large Language Models Encode Clinical Knowledgecited by 1,807
  2. 2023A Survey on Evaluation of Large Language Modelscited by 1,227
  3. 2023Translating Radiology Reports into Plain Language using ChatGPT and GPT-4 with Prompt Learning: Promising Results, Limitations, and Potentialcited by 39

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 6; 2025: 4; 2026: 3.

13 papers.

Sort:

  1. 2024A Framework for Human Evaluation of Large Language Models in Healthcare Derived from Literature Reviewcited by 65
  2. 2024Anchoring Bias in Large Language Models: An Experimental Studycited by 37
  3. 2025Digital Companionship: Overlapping Uses of AI Companions and AI Assistantscited by 7
  4. 2024Empowering Meta-Analysis: Leveraging Large Language Models for Scientific Synthesiscited by 4
  5. 2024Retrospective Comparative Analysis of Prostate Cancer In-Basket Messages: Responses from Closed-Domain LLM vs. Clinical Teamscited by 2
  6. 2024Language Models And A Second Opinion Use Case: The Pocket Professionalcited by 1
  7. 2026Authority Signals in AI Cited Health Sources: A Framework for Evaluating Source Credibility in ChatGPT Responsescited by 1
  8. 2024Performance Evaluation of Lightweight Open-source Large Language Models in Pediatric Consultations: A Comparative Analysiscited by 0
  9. 2025Human-Centered AI in Multidisciplinary Medical Discussions: Evaluating the Feasibility of a Chat-Based Approach to Case Assessmentcited by 0
  10. 2025LaQual: An Automated Framework for LLM App Quality Evaluationcited by 0
  11. 2025COMPARE: Clinical Optimization with Modular Planning and Assessment via RAG-Enhanced AI-OCT: Superior Decision Support for Percutaneous Coronary Intervention Compared to ChatGPT-5 and Junior Operatorscited by 0
  12. 2026Preferred, Not Safer: Pairwise Preference Is a Poor Proxy for Clinical Safetycited by 0
1 more
  1. 2026Performance of a domain-specific large language model in answering patient questions in psychiatrycited by 0