# Medically Aware GPT-3 as a Data Generator for Medical Dialogue Summarization

arXiv: https://arxiv.org/abs/2110.07356 · HTML: https://mlsanity.com/paper/2110.07356/

## Key facts

- Cited by 76 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 28 references that are in the map.
- First posted on arXiv 9 Sep 2021.

## Builds on (most cited first, 19 of 19)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019)
- [Get To The Point: Summarization with Pointer-Generator Networks](https://mlsanity.com/paper/1704.04368/) (2017)
- [What Makes Good In-Context Examples for GPT-3?](https://mlsanity.com/paper/2101.06804/) (2021)
- [PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization](https://mlsanity.com/paper/1912.08777/) (2019)
- [Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyond](https://mlsanity.com/paper/1602.06023/) (2016)
- [Recipes for building an open-domain chatbot](https://mlsanity.com/paper/2004.13637/) (2020)
- [Evaluating the Factual Consistency of Abstractive Text Summarization](https://mlsanity.com/paper/1910.12840/) (2019)
- [SummaRuNNer: A Recurrent Neural Network based Sequence Model for Extractive Summarization of Documents](https://mlsanity.com/paper/1611.04230/) (2016)
- [ProphetNet: Predicting Future N-gram for Sequence-to-Sequence Pre-training](https://mlsanity.com/paper/2001.04063/) (2020)
- [Incorporating Copying Mechanism in Sequence-to-Sequence Learning](https://mlsanity.com/paper/1603.06393/) (2016)
- [Learning to Summarize Radiology Findings](https://mlsanity.com/paper/1809.04698/) (2018)
- [Dr. Summarize: Global Summarization of Medical Dialogue by Exploiting Local Structures](https://mlsanity.com/paper/2009.08666/) (2020)
- [Abstractive Dialogue Summarization with Sentence-Gated Modeling Optimized by Dialogue Acts](https://mlsanity.com/paper/1809.05715/) (2018)
- [Topic-aware Pointer-Generator Networks for Summarizing Spoken Conversations](https://mlsanity.com/paper/1910.01335/) (2019)
- [Extractive Summarization of EHR Discharge Notes](https://mlsanity.com/paper/1810.12085/) (2018)
- [Extracting Structured Data from Physician-Patient Conversations By Predicting Noteworthy Utterances](https://mlsanity.com/paper/2007.07151/) (2020)
- [Clinical Concept Extraction: a Methodology Review](https://mlsanity.com/paper/1910.11377/) (2019)

## Built on by (most cited first, 30 of 76)

- [A Comprehensive Survey on Process-Oriented Automatic Text Summarization with Exploration of LLM-Based Methods](https://mlsanity.com/paper/2403.02901/) (2024)
- [A Survey of GPT-3 Family Large Language Models Including ChatGPT and GPT-4](https://mlsanity.com/paper/2310.12321/) (2023)
- [An Empirical Survey on Long Document Summarization: Datasets, Models and Metrics](https://mlsanity.com/paper/2207.00939/) (2022)
- [NL2TL: Transforming Natural Languages to Temporal Logics using Large Language Models](https://mlsanity.com/paper/2305.07766/) (2023)
- [DERA: Enhancing Large Language Model Completions with Dialog-Enabled Resolving Agents](https://mlsanity.com/paper/2303.17071/) (2023)
- [Human Evaluation and Correlation with Automatic Metrics in Consultation Note Generation](https://mlsanity.com/paper/2204.00447/) (2022)
- [PriMock57: A Dataset Of Primary Care Mock Consultations](https://mlsanity.com/paper/2204.00333/) (2022)
- [A Survey of Small Language Models](https://mlsanity.com/paper/2410.20011/) (2024)
- [Data Augmentation using Large Language Models: Data Perspectives, Learning Paradigms and Challenges](https://mlsanity.com/paper/2403.02990/) (2024)
- [The Shaky Foundations of Clinical Foundation Models: A Survey of Large Language Models and Foundation Models for EMRs](https://mlsanity.com/paper/2303.12961/) (2023)
- [Two Directions for Clinical Data Generation with Large Language Models: Data-to-Label and Label-to-Data](https://mlsanity.com/paper/2401.06774/) (2023)
- [MediTOD: An English Dialogue Dataset for Medical History Taking with Comprehensive Annotations](https://mlsanity.com/paper/2410.14204/) (2024)
- [Retrieval augmentation of large language models for lay language generation](https://mlsanity.com/paper/2211.03818/) (2022)
- [IC3: Image Captioning by Committee Consensus](https://mlsanity.com/paper/2302.01328/) (2023)
- [Knowledge-Infused Prompting: Assessing and Advancing Clinical Text Data Generation with Large Language Models](https://mlsanity.com/paper/2311.00287/) (2023)
- [SynthBio: A Case Study in Human-AI Collaborative Curation of Text Datasets](https://mlsanity.com/paper/2111.06467/) (2021)
- [Taking the Next Step with Generative Artificial Intelligence: The Transformative Role of Multimodal Large Language Models in Science Education](https://mlsanity.com/paper/2401.00832/) (2024)
- [See What LLMs Cannot Answer: A Self-Challenge Framework for Uncovering LLM Weaknesses](https://mlsanity.com/paper/2408.08978/) (2024)
- [Natural Language Processing for Smart Healthcare](https://mlsanity.com/paper/2110.15803/) (2021)
- [A Survey on Medical Document Summarization](https://mlsanity.com/paper/2212.01669/) (2022)
- [User-Driven Research of Medical Note Generation Software](https://mlsanity.com/paper/2205.02549/) (2022)
- [SummQA at MEDIQA-Chat 2023:In-Context Learning with GPT-4 for Medical Summarization](https://mlsanity.com/paper/2306.17384/) (2023)
- [Exploring the Efficacy of Large Language Models in Summarizing Mental Health Counseling Sessions: A Benchmark Study](https://mlsanity.com/paper/2402.19052/) (2024)
- [Can LLMs Replace Economic Choice Prediction Labs? The Case of Language-based Persuasion Games](https://mlsanity.com/paper/2401.17435/) (2024)
- [Knowledge Distillation Using Frontier Open-source LLMs: Generalizability and the Role of Synthetic Data](https://mlsanity.com/paper/2410.18588/) (2024)
- [a survey on GPT-3](https://mlsanity.com/paper/2212.00857/) (2022)
- [Conversational AI-Powered Design: ChatGPT as Designer, User, and Product](https://mlsanity.com/paper/2302.07406/) (2023)
- [What are the Desired Characteristics of Calibration Sets? Identifying Correlates on Long Form Scientific Summarization](https://mlsanity.com/paper/2305.07615/) (2023)
- [Experience and Evidence are the eyes of an excellent summarizer! Towards Knowledge Infused Multi-modal Clinical Conversation Summarization](https://mlsanity.com/paper/2309.15739/) (2023)
- [WangLab at MEDIQA-Chat 2023: Clinical Note Generation from Doctor-Patient Conversations using Large Language Models](https://mlsanity.com/paper/2305.02220/) (2023)

## Abstract (arXiv)

In medical dialogue summarization, summaries must be coherent and must capture all the medically relevant information in the dialogue. However, learning effective models for summarization require large amounts of labeled data which is especially hard to obtain. We present an algorithm to create synthetic training data with an explicit focus on capturing medically relevant information. We utilize GPT-3 as the backbone of our algorithm and scale 210 human labeled examples to yield results comparable to using 6400 human labeled examples (~30x) leveraging low-shot learning and an ensemble method. In detailed experiments, we show that this approach produces high quality training data that can further be combined with human labeled data to get summaries that are strongly preferable to those produced by models trained on human data alone both in terms of medical accuracy and coherency.