# Summarizing, Simplifying, and Synthesizing Medical Evidence Using GPT-3 (with Varying Success)

arXiv: https://arxiv.org/abs/2305.06299 · HTML: https://mlsanity.com/paper/2305.06299/

## Key facts

- Cited by 40 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 15 references that are in the map.
- First posted on arXiv 10 May 2023.

## Builds on (most cited first, 7 of 7)

- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [Large Language Models Encode Clinical Knowledge](https://mlsanity.com/paper/2212.13138/) (2022)
- [SummEval: Re-evaluating Summarization Evaluation](https://mlsanity.com/paper/2007.12626/) (2020)
- [News Summarization and Evaluation in the Era of GPT-3](https://mlsanity.com/paper/2209.12356/) (2022)
- [Can large language models reason about medical questions?](https://mlsanity.com/paper/2207.08143/) (2022)
- [Can Language Models be Biomedical Knowledge Bases?](https://mlsanity.com/paper/2109.07154/) (2021)
- [GENIE: Toward Reproducible and Standardized Human Evaluation for Text Generation](https://mlsanity.com/paper/2101.06561/) (2021)

## Built on by (most cited first, 30 of 40)

- [MedAgents: Large Language Models as Collaborators for Zero-shot Medical Reasoning](https://mlsanity.com/paper/2311.10537/) (2023)
- [MiniCheck: Efficient Fact-Checking of LLMs on Grounding Documents](https://mlsanity.com/paper/2404.10774/) (2024)
- [RECOMP: Improving Retrieval-Augmented LMs with Compression and Selective Augmentation](https://mlsanity.com/paper/2310.04408/) (2023)
- [A Survey of Large Language Models in Medicine: Progress, Application, and Challenge](https://mlsanity.com/paper/2311.05112/) (2023)
- [MedCalc-Bench: Evaluating Large Language Models for Medical Calculations](https://mlsanity.com/paper/2406.12036/) (2024)
- [Improving Retrieval-Augmented Generation in Medicine with Iterative Follow-up Questions](https://mlsanity.com/paper/2408.00727/) (2024)
- [A Systematic Survey of Text Summarization: From Statistical Methods to Large Language Models](https://mlsanity.com/paper/2406.11289/) (2024)
- [SentimentGPT: Exploiting GPT for Advanced Sentiment Analysis and its Departure from Current Machine Learning](https://mlsanity.com/paper/2307.10234/) (2023)
- [MedSyn: LLM-based Synthetic Medical Text Generation Framework](https://mlsanity.com/paper/2408.02056/) (2024)
- [Automated Metrics for Medical Multi-Document Summarization Disagree with Human Evaluations](https://mlsanity.com/paper/2305.13693/) (2023)
- [FactPICO: Factuality Evaluation for Plain Language Summarization of Medical Evidence](https://mlsanity.com/paper/2402.11456/) (2024)
- [PubMed and Beyond: Biomedical Literature Search in the Age of Artificial Intelligence](https://mlsanity.com/paper/2307.09683/) (2023)
- [Accelerating Clinical Evidence Synthesis with Large Language Models](https://mlsanity.com/paper/2406.17755/) (2024)
- [InfoLossQA: Characterizing and Recovering Information Loss in Text Simplification](https://mlsanity.com/paper/2401.16475/) (2024)
- [Multilingual Simplification of Medical Texts](https://mlsanity.com/paper/2305.12532/) (2023)
- [Unlocking the Power of Spatial and Temporal Information in Medical Multimodal Pre-training](https://mlsanity.com/paper/2405.19654/) (2024)
- [Using Natural Language Explanations to Rescale Human Judgments](https://mlsanity.com/paper/2305.14770/) (2023)
- [Large Language Models on Wikipedia-Style Survey Generation: an Evaluation in NLP Concepts](https://mlsanity.com/paper/2308.10410/) (2023)
- [KIWI: A Dataset of Knowledge-Intensive Writing Instructions for Answering Research Questions](https://mlsanity.com/paper/2403.03866/) (2024)
- [A foundation model for human-AI collaboration in medical literature mining](https://mlsanity.com/paper/2501.16255/) (2025)
- [From Scores to Steps: Diagnosing and Improving LLM Performance in Evidence-Based Medical Calculations](https://mlsanity.com/paper/2509.16584/) (2025)
- [Evaluating the Factuality of Zero-shot Summarizers Across Varied Domains](https://mlsanity.com/paper/2402.03509/) (2024)
- [A Perspective for Adapting Generalist AI to Specialized Medical AI Applications and Their Challenges](https://mlsanity.com/paper/2411.00024/) (2024)
- [Elucidating Mechanisms of Demographic Bias in LLMs for Healthcare](https://mlsanity.com/paper/2502.13319/) (2025)
- [Can Large Language Models Match the Conclusions of Systematic Reviews?](https://mlsanity.com/paper/2505.22787/) (2025)
- [Evaluating Large Language Models for Public Health Classification and Extraction Tasks](https://mlsanity.com/paper/2405.14766/) (2024)
- [It Cannot Be Right If It Was Written by AI: On Lawyers' Preferences of Documents Perceived as Authored by an LLM vs a Human](https://mlsanity.com/paper/2407.06798/) (2024)
- [Relabeling Minimal Training Subset to Flip a Prediction](https://mlsanity.com/paper/2305.12809/) (2023)
- [LitSumm: Large language models for literature summarisation of non-coding RNAs](https://mlsanity.com/paper/2311.03056/) (2023)
- [Leveraging Generative AI for Clinical Evidence Summarization Needs to Ensure Trustworthiness](https://mlsanity.com/paper/2311.11211/) (2023)

## Abstract (arXiv)

Large language models, particularly GPT-3, are able to produce high quality summaries of general domain news articles in few- and zero-shot settings. However, it is unclear if such models are similarly capable in more specialized, high-stakes domains such as biomedicine. In this paper, we enlist domain experts (individuals with medical training) to evaluate summaries of biomedical articles generated by GPT-3, given zero supervision. We consider both single- and multi-document settings. In the former, GPT-3 is tasked with generating regular and plain-language summaries of articles describing randomized controlled trials; in the latter, we assess the degree to which GPT-3 is able to synthesize evidence reported across a collection of articles. We design an annotation scheme for evaluating model outputs, with an emphasis on assessing the factual accuracy of generated summaries. We find that while GPT-3 is able to summarize and simplify single biomedical articles faithfully, it struggles to provide accurate aggregations of findings over multiple documents. We release all data and annotations used in this work.