mlsanityData to 7 Oct 2026

Are Large Language Models Ready for Healthcare? A Comparative Study on Clinical Language Understanding

arXiv 2304.05368 ↗First posted 9 Apr 2023cs.CL cs.AIcited by 40

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) have made significant progress in various domains, including healthcare. However, the specialized nature of clinical language understanding tasks presents unique challenges and limitations that warrant further investigation. In this study, we conduct a comprehensive evaluation of state-of-the-art LLMs, namely GPT-3.5, GPT-4, and Bard, within the realm of clinical language understanding tasks. These tasks span a diverse range, including named entity recognition, relation extraction, natural language inference, semantic textual similarity, document classification, and question-answering. We also introduce a novel prompting strategy, self-questioning prompting (SQP), tailored to enhance LLMs' performance by eliciting informative questions and answers pertinent to the clinical scenarios at hand. Our evaluation underscores the significance of task-specific learning strategies and prompting techniques for improving LLMs' effectiveness in healthcare-related tasks. Additionally, our in-depth error analysis on the challenging relation extraction task offers valuable insights into error distribution and potential avenues for improvement using SQP. Our study sheds light on the practical implications of employing LLMs in the specialized domain of healthcare, serving as a foundation for future research and the development of potential applications in healthcare settings.

Key facts. Cited by 40 papers in the map (data through 7 Oct 2026). Builds on 17 of its 51 references that are in the map. First posted on arXiv 9 Apr 2023.

Builds on

17 of 51 references.

References · 17

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  6. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  7. 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,935
  8. 2022Large Language Models Encode Clinical Knowledgecited by 1,807
  9. 2019Publicly Available Clinical BERT Embeddingscited by 995
  10. 2023Capabilities of GPT-4 on Medical Challenge Problemscited by 630
  11. 2023Is ChatGPT a Good NLG Evaluator? A Preliminary Studycited by 420
  12. 2019Transfer Learning in Biomedical Natural Language Processing: An Evaluation of BERT and ELMo on Ten Benchmarking Datasetscited by 298
5 more
  1. 2023Can ChatGPT Understand Too? A Comparative Study on ChatGPT and Fine-tuned BERTcited by 155
  2. 2018Lessons from Natural Language Inference in the Clinical Domaincited by 149
  3. 2022Iteratively Prompt Pre-trained Language Models for Chain of Thoughtcited by 75
  4. 2022Integrating Physiological Time Series and Clinical Notes with Transformer for Early Prediction of Sepsiscited by 14
  5. 2020Comparative Analysis of Text Classification Approaches in Electronic Health Recordscited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 13; 2024: 18; 2025: 6; 2026: 3.

40 papers.

Sort:

By area:

  1. 2023Large language models in healthcare and medical domain: A reviewcited by 126
  2. 2023A Survey of Large Language Models in Medicine: Progress, Application, and Challengecited by 123
  3. 2023The Impact of Large Language Models on Scientific Discovery: a Preliminary Study using GPT-4cited by 112
  4. 2023GPT4GEO: How a Language Model Sees the World's Geographycited by 68
  5. 2024Risk Taxonomy, Mitigation, and Assessment Benchmarks of Large Language Model Systemscited by 63
  6. 2023Metacognitive Prompting Improves Understanding in Large Language Modelscited by 58
  7. 2023TRAM: Benchmarking Temporal Reasoning for Large Language Modelscited by 39
  8. 2024A Survey on Medical Large Language Models: Technology, Application, Trustworthiness, and Future Directionscited by 34
  9. 2024Bias patterns in the application of LLMs for clinical decision support: A comprehensive studycited by 27
  10. 2023An Empirical Study on the Robustness of the Segment Anything Model (SAM)cited by 23
  11. 2024ClinicalMamba: A Generative Clinical Language Model on Longitudinal Clinical Notescited by 22
  12. 2023User Modeling in the Era of Large Language Models: Current Research and Future Directionscited by 18
28 more
  1. 2023Gemini in Reasoning: Unveiling Commonsense in Multimodal Large Language Modelscited by 15
  2. 2024Logical Consistency of Large Language Models in Fact-checkingcited by 15
  3. 2024Unlocking the Power of Spatial and Temporal Information in Medical Multimodal Pre-trainingcited by 13
  4. 2023Evaluating Large Language Models for Radiology Natural Language Processingcited by 12
  5. 2024From Explainable to Interpretable Deep Learning for Natural Language Processing in Healthcare: How Far from Reality?cited by 11
  6. 2023Large Language Models Illuminate a Progressive Pathway to Artificial Healthcare Assistant: A Reviewcited by 8
  7. 2024Generalist embedding models are better at short-context clinical semantic search than specialized embedding modelscited by 7
  8. 2025LLM-based Semantic Augmentation for Harmful Content Detectioncited by 7
  9. 2024FairEHR-CLP: Towards Fairness-Aware Clinical Predictions with Contrastive Learning in Multimodal Electronic Health Recordscited by 6
  10. 2023GPT vs Human for Scientific Reviews: A Dual Source Review on Applications of ChatGPT in Sciencecited by 5
  11. 2024Unveiling and Mitigating Bias in Mental Health Analysis with Large Language Modelscited by 5
  12. 2024PediaBench: A Comprehensive Chinese Pediatric Dataset for Benchmarking Large Language Modelscited by 5
  13. 2024Unlocking LLMs: Addressing Scarce Data and Bias Challenges in Mental Healthcited by 5
  14. 2025Are Large Language Models Good at Detecting Propaganda?cited by 5
  15. 2024BioMistral-NLU: Towards More Generalizable Medical Language Understanding through Instruction Tuningcited by 4
  16. 2025KScope: A Framework for Characterizing the Knowledge Status of Language Modelscited by 3
  17. 2025MedKGEval: A Knowledge Graph-Based Multi-Turn Evaluation Framework for Open-Ended Patient Interactions with Clinical LLMscited by 3
  18. 2024Clinical information extraction for Low-resource languages with Few-shot learning using Pre-trained language models and Promptingcited by 2
  19. 2024On the Reliability of Large Language Models to Misinformed and Demographically-Informed Promptscited by 2
  20. 2025A Benchmark for End-to-End Zero-Shot Biomedical Relation Extraction with LLMs: Experiments with OpenAI Modelscited by 2
  21. 2026Illusions of Confidence? Diagnosing LLM Truthfulness via Neighborhood Consistencycited by 2
  22. 2023Speculative Exploration on the Concept of Artificial Agents Conducting Autonomous Researchcited by 1
  23. 2024Reinforcement of Explainability of ChatGPT Prompts by Embedding Breast Cancer Self-Screening Rules into AI Responsescited by 1
  24. 2024The Role of Language Models in Modern Healthcare: A Comprehensive Reviewcited by 1
  25. 2024Using LLMs to label medical papers according to the CIViC evidence modelcited by 0
  26. 2025Performance and Practical Considerations of Large and Small Language Models in Clinical Decision Support in Rheumatologycited by 0
  27. 2026Relation Extraction Capabilities of LLMs on Clinical Text: A Bilingual Evaluation for English and Turkishcited by 0
  28. 2026Bridging the English-Arabic Medical Knowledge Gap: Targeted Low-Rank Adaptation via Causal Layer Selectioncited by 0