# SummQA at MEDIQA-Chat 2023:In-Context Learning with GPT-4 for Medical Summarization

arXiv: https://arxiv.org/abs/2306.17384 · HTML: https://mlsanity.com/paper/2306.17384/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 48 references that are in the map.
- First posted on arXiv 30 Jun 2023.

## Builds on (most cited first, 24 of 24)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [BioBERT: a pre-trained biomedical language representation model for biomedical text mining](https://mlsanity.com/paper/1901.08746/) (2019)
- [MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers](https://mlsanity.com/paper/2002.10957/) (2020)
- [Calibrate Before Use: Improving Few-Shot Performance of Language Models](https://mlsanity.com/paper/2102.09690/) (2021)
- [Making Pre-trained Language Models Better Few-shot Learners](https://mlsanity.com/paper/2012.15723/) (2020)
- [Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity](https://mlsanity.com/paper/2104.08786/) (2021)
- [Learning To Retrieve Prompts for In-Context Learning](https://mlsanity.com/paper/2112.08633/) (2021)
- [Capabilities of GPT-4 on Medical Challenge Problems](https://mlsanity.com/paper/2303.13375/) (2023)
- [True Few-Shot Learning with Language Models](https://mlsanity.com/paper/2105.11447/) (2021)
- [News Summarization and Evaluation in the Era of GPT-3](https://mlsanity.com/paper/2209.12356/) (2022)
- [Newsroom: A Dataset of 1.3 Million Summaries with Diverse Extractive Strategies](https://mlsanity.com/paper/1804.11283/) (2018)
- [ACI-BENCH: a Novel Ambient Clinical Intelligence Dataset for Benchmarking Automatic Visit Note Generation](https://mlsanity.com/paper/2306.02022/) (2023)
- [Medically Aware GPT-3 as a Data Generator for Medical Dialogue Summarization](https://mlsanity.com/paper/2110.07356/) (2021)
- [Complementary Explanations for Effective In-Context Learning](https://mlsanity.com/paper/2211.13892/) (2022)
- [Prompt Consistency for Zero-Shot Task Generalization](https://mlsanity.com/paper/2205.00049/) (2022)
- [Dr. Summarize: Global Summarization of Medical Dialogue by Exploiting Local Structures](https://mlsanity.com/paper/2009.08666/) (2020)
- [Multi-document Summarization with Maximal Marginal Relevance-guided Reinforcement Learning](https://mlsanity.com/paper/2010.00117/) (2020)
- [Graph-Evolving Meta-Learning for Low-Resource Medical Dialogue Generation](https://mlsanity.com/paper/2012.11988/) (2020)
- [A Survey on Medical Document Summarization](https://mlsanity.com/paper/2212.01669/) (2022)
- [He Said, She Said: Style Transfer for Shifting the Perspective of Dialogues](https://mlsanity.com/paper/2210.15462/) (2022)
- [Generating Query Focused Summaries without Fine-tuning the Transformer-based Pre-trained Models](https://mlsanity.com/paper/2303.06230/) (2023)

## Built on by (most cited first, 13 of 13)

- [Adapted Large Language Models Can Outperform Medical Experts in Clinical Text Summarization](https://mlsanity.com/paper/2309.07430/) (2023)
- [Membership Inference Attacks Against In-Context Learning](https://mlsanity.com/paper/2409.01380/) (2024)
- [Exploring Robustness in Doctor-Patient Conversation Summarization: An Analysis of Out-of-Domain SOAP Notes](https://mlsanity.com/paper/2406.02826/) (2024)
- [ContextLeak: Auditing Leakage in Private In-Context Learning Methods](https://mlsanity.com/paper/2512.16059/) (2025)
- [Enhancing Summarization Performance through Transformer-Based Prompt Engineering in Automated Medical Reporting](https://mlsanity.com/paper/2311.13274/) (2023)
- [TN-Eval: Rubric and Evaluation Protocols for Measuring the Quality of Behavioral Therapy Notes](https://mlsanity.com/paper/2503.20648/) (2025)
- [Deciphering Diagnoses: How Large Language Models Explanations Influence Clinical Decision Making](https://mlsanity.com/paper/2310.01708/) (2023)
- [Leveraging language models for summarizing mental state examinations: A comprehensive evaluation and dataset release](https://mlsanity.com/paper/2403.20145/) (2024)
- [FactsR: A Safer Method for Producing High Quality Healthcare Documentation](https://mlsanity.com/paper/2505.10360/) (2025)
- [The use of large language models to enhance cancer clinical trial educational materials](https://mlsanity.com/paper/2412.01955/) (2024)
- [Leveraging Large Language Models for Rare Disease Named Entity Recognition](https://mlsanity.com/paper/2508.09323/) (2025)
- [Membership Inference Attacks for Retrieval Based In-Context Learning for Document Question Answering](https://mlsanity.com/paper/2605.04116/) (2026)
- [Knowledge Graph-Augmented Ambient AI for Clinical Note Generation](https://mlsanity.com/paper/2609.22239/) (2026)

## Abstract (arXiv)

Medical dialogue summarization is challenging due to the unstructured nature of medical conversations, the use of medical terminology in gold summaries, and the need to identify key information across multiple symptom sets. We present a novel system for the Dialogue2Note Medical Summarization tasks in the MEDIQA 2023 Shared Task. Our approach for section-wise summarization (Task A) is a two-stage process of selecting semantically similar dialogues and using the top-k similar dialogues as in-context examples for GPT-4. For full-note summarization (Task B), we use a similar solution with k=1. We achieved 3rd place in Task A (2nd among all teams), 4th place in Task B Division Wise Summarization (2nd among all teams), 15th place in Task A Section Header Classification (9th among all teams), and 8th place among all teams in Task B. Our results highlight the effectiveness of few-shot prompting for this task, though we also identify several weaknesses of prompting-based approaches. We compare GPT-4 performance with several finetuned baselines. We find that GPT-4 summaries are more abstractive and shorter. We make our code publicly available.