# Evaluation of GPT-3.5 and GPT-4 for supporting real-world information needs in healthcare delivery

arXiv: https://arxiv.org/abs/2304.13714 · HTML: https://mlsanity.com/paper/2304.13714/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 42 references that are in the map.
- First posted on arXiv 26 Apr 2023.

## Builds on (most cited first, 18 of 18)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [Large Language Models Encode Clinical Knowledge](https://mlsanity.com/paper/2212.13138/) (2022)
- [SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems](https://mlsanity.com/paper/1905.00537/) (2019)
- [What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Exams](https://mlsanity.com/paper/2009.13081/) (2020)
- [Capabilities of GPT-4 on Medical Challenge Problems](https://mlsanity.com/paper/2303.13375/) (2023)
- [Benchmarking Large Language Models for News Summarization](https://mlsanity.com/paper/2301.13848/) (2023)
- [A Comprehensive Survey on Pretrained Foundation Models: A History from BERT to ChatGPT](https://mlsanity.com/paper/2302.09419/) (2023)
- [AI and the Everything in the Whole Wide World Benchmark](https://mlsanity.com/paper/2111.15366/) (2021)
- [ChatGPT for Robotics: Design Principles and Model Abilities](https://mlsanity.com/paper/2306.17582/) (2023)
- [Can large language models reason about medical questions?](https://mlsanity.com/paper/2207.08143/) (2022)
- [Compute Trends Across Three Eras of Machine Learning](https://mlsanity.com/paper/2202.05924/) (2022)
- [Prompting PaLM for Translation: Assessing Strategies and Performance](https://mlsanity.com/paper/2211.09102/) (2022)
- [ChatGPT Makes Medicine Easy to Swallow: An Exploratory Case Study on Simplified Radiology Reports](https://mlsanity.com/paper/2212.14882/) (2022)
- [Large AI Models in Health Informatics: Applications, Challenges, and the Future](https://mlsanity.com/paper/2303.11568/) (2023)
- [The Shaky Foundations of Clinical Foundation Models: A Survey of Large Language Models and Foundation Models for EMRs](https://mlsanity.com/paper/2303.12961/) (2023)

## Built on by (most cited first, 30 of 30)

- [HealthBench: Evaluating Large Language Models Towards Improved Human Health](https://mlsanity.com/paper/2505.08775/) (2025)
- [What can Large Language Models do in chemistry? A comprehensive benchmark on eight tasks](https://mlsanity.com/paper/2305.18365/) (2023)
- [PaperQA: Retrieval-Augmented Generative Agent for Scientific Research](https://mlsanity.com/paper/2312.07559/) (2023)
- [Do Language Models Know When They're Hallucinating References?](https://mlsanity.com/paper/2305.18248/) (2023)
- [ClashEval: Quantifying the tug-of-war between an LLM's internal prior and external evidence](https://mlsanity.com/paper/2404.10198/) (2024)
- [ExpertQA: Expert-Curated Questions and Attributed Answers](https://mlsanity.com/paper/2309.07852/) (2023)
- [Large Language Models are Geographically Biased](https://mlsanity.com/paper/2402.02680/) (2024)
- [Cognitive Mirage: A Review of Hallucinations in Large Language Models](https://mlsanity.com/paper/2309.06794/) (2023)
- [GPT4GEO: How a Language Model Sees the World's Geography](https://mlsanity.com/paper/2306.00020/) (2023)
- [A Computational Framework for Behavioral Assessment of LLM Therapists](https://mlsanity.com/paper/2401.00820/) (2024)
- [Publicly Shareable Clinical Large Language Model Built on Synthetic Clinical Notes](https://mlsanity.com/paper/2309.00237/) (2023)
- [SciFIBench: Benchmarking Large Multimodal Models for Scientific Figure Interpretation](https://mlsanity.com/paper/2405.08807/) (2024)
- [BioPlanner: Automatic Evaluation of LLMs on Protocol Planning in Biology](https://mlsanity.com/paper/2310.10632/) (2023)
- [History, Development, and Principles of Large Language Models-An Introductory Survey](https://mlsanity.com/paper/2402.06853/) (2024)
- [How well do LLMs cite relevant medical references? An evaluation framework and analyses](https://mlsanity.com/paper/2402.02008/) (2024)
- [Mitigating the Risk of Health Inequity Exacerbated by Large Language Models](https://mlsanity.com/paper/2410.05180/) (2024)
- [Tackling Bias in Pre-trained Language Models: Current Trends and Under-represented Societies](https://mlsanity.com/paper/2312.01509/) (2023)
- [CheX-GPT: Harnessing Large Language Models for Enhanced Chest X-ray Report Labeling](https://mlsanity.com/paper/2401.11505/) (2024)
- [Elucidating Mechanisms of Demographic Bias in LLMs for Healthcare](https://mlsanity.com/paper/2502.13319/) (2025)
- [Insights into Classifying and Mitigating LLMs' Hallucinations](https://mlsanity.com/paper/2311.08117/) (2023)
- [RACER: An LLM-powered Methodology for Scalable Analysis of Semi-structured Mental Health Interviews](https://mlsanity.com/paper/2402.02656/) (2024)
- [Zero-shot Learning with Minimum Instruction to Extract Social Determinants and Family History from Clinical Notes using GPT Model](https://mlsanity.com/paper/2309.05475/) (2023)
- [Beyond Accuracy: Investigating Error Types in GPT-4 Responses to USMLE Questions](https://mlsanity.com/paper/2404.13307/) (2024)
- [Large Language Models as Agents in the Clinic](https://mlsanity.com/paper/2309.10895/) (2023)
- [Interactive Multi-fidelity Learning for Cost-effective Adaptation of Language Model with Sparse Human Supervision](https://mlsanity.com/paper/2310.20153/) (2023)
- [MedConsultBench: A Full-Cycle, Fine-Grained, Process-Aware Benchmark for Medical Consultation Agents](https://mlsanity.com/paper/2601.12661/) (2026)
- [Answering real-world clinical questions using large language model based systems](https://mlsanity.com/paper/2407.00541/) (2024)
- [Crossing New Frontiers: Knowledge-Augmented Large Language Model Prompting for Zero-Shot Text-Based De Novo Molecule Design](https://mlsanity.com/paper/2408.11866/) (2024)
- [Red Teaming Large Language Models for Healthcare](https://mlsanity.com/paper/2505.00467/) (2025)
- [Attaining Human`s Desirable Outcomes in Human-AI Interaction via Structural Causal Games](https://mlsanity.com/paper/2405.16588/) (2024)

## Abstract (arXiv)

Despite growing interest in using large language models (LLMs) in healthcare, current explorations do not assess the real-world utility and safety of LLMs in clinical settings. Our objective was to determine whether two LLMs can serve information needs submitted by physicians as questions to an informatics consultation service in a safe and concordant manner. Sixty six questions from an informatics consult service were submitted to GPT-3.5 and GPT-4 via simple prompts. 12 physicians assessed the LLM responses' possibility of patient harm and concordance with existing reports from an informatics consultation service. Physician assessments were summarized based on majority vote. For no questions did a majority of physicians deem either LLM response as harmful. For GPT-3.5, responses to 8 questions were concordant with the informatics consult report, 20 discordant, and 9 were unable to be assessed. There were 29 responses with no majority on "Agree", "Disagree", and "Unable to assess". For GPT-4, responses to 13 questions were concordant, 15 discordant, and 3 were unable to be assessed. There were 35 responses with no majority. Responses from both LLMs were largely devoid of overt harm, but less than 20% of the responses agreed with an answer from an informatics consultation service, responses contained hallucinated references, and physicians were divided on what constitutes harm. These results suggest that while general purpose LLMs are able to provide safe and credible responses, they often do not meet the specific information need of a given question. A definitive evaluation of the usefulness of LLMs in healthcare settings will likely require additional research on prompt engineering, calibration, and custom-tailoring of general purpose models.