mlsanityData to 7 Oct 2026

Large Language Models are Few-Shot Clinical Information Extractors

arXiv 2205.12689 ↗First posted 25 May 2022cs.CL cs.AIcited by 206

Not on the map. Why?

Abstract (arXiv)

A long-running goal of the clinical NLP community is the extraction of important variables trapped in clinical notes. However, roadblocks have included dataset shift from the general domain and a lack of public clinical corpora and annotations. In this work, we show that large language models, such as InstructGPT, perform well at zero- and few-shot information extraction from clinical text despite not being trained specifically for the clinical domain. Whereas text classification and generation performance have already been studied extensively in such models, here we additionally demonstrate how to leverage them to tackle a diverse set of NLP tasks which require more structured outputs, including span identification, token-level sequence classification, and relation extraction. Further, due to the dearth of available data to evaluate these systems, we introduce new datasets for benchmarking few-shot clinical information extraction based on a manual re-annotation of the CASI dataset for new tasks. On the clinical extraction tasks we studied, the GPT-3 systems significantly outperform existing zero- and few-shot baselines.

Key facts. Cited by 206 papers in the map (data through 7 Oct 2026). Builds on 45 of its 85 references that are in the map. First posted on arXiv 25 May 2022.

Builds on

45 of 85 references.

References · 45

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  5. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  6. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,462
  7. 2018Deep contextualized word representationscited by 2,285
  8. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  9. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  10. 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,935
  11. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  12. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,413
33 more
  1. 2019CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparisoncited by 1,410
  2. 2020Domain-Specific Language Model Pretraining for Biomedical Natural Language Processingcited by 996
  3. 2019Publicly Available Clinical BERT Embeddingscited by 995
  4. 2020Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inferencecited by 934
  5. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 886
  6. 2020Stanza: A Python Natural Language Processing Toolkit for Many Human Languagescited by 719
  7. 2017Snorkel: Rapid Training Data Creation with Weak Supervisioncited by 370
  8. 2021AI Chains: Transparent and Controllable Human-AI Interaction by Chaining Large Language Model Promptscited by 265
  9. 2019ScispaCy: Fast and Robust Models for Biomedical Natural Language Processingcited by 244
  10. 2017End-to-end Neural Coreference Resolutioncited by 240
  11. 2019A Unified MRC Framework for Named Entity Recognitioncited by 199
  12. 2021Want To Reduce Labeling Cost? GPT-3 Can Helpcited by 196
  13. 2021Reframing Instructional Prompts to GPTk's Languagecited by 176
  14. 2019Simple BERT Models for Relation Extraction and Semantic Role Labelingcited by 154
  15. 2021Template-Based Named Entity Recognition Using BARTcited by 153
  16. 2022Instruction Induction: From Few Examples to Natural Language Task Descriptionscited by 152
  17. 2018Construction of the Literature Graph in Semantic Scholarcited by 118
  18. 2019Entity-Relation Extraction as Multi-Turn Question Answeringcited by 96
  19. 2021WRENCH: A Comprehensive Benchmark for Weak Supervisioncited by 95
  20. 2022Thinking about GPT-3 In-Context Learning for Biomedical IE? Think Againcited by 83
  21. 2018A Corpus with Multi-Level Annotations of Patients, Interventions and Outcomes to Support Language Processing for Medical Literaturecited by 79
  22. 2022Language Models in the Loop: Incorporating Prompting into Weak Supervisioncited by 51
  23. 2020BERT-XML: Large Scale Automated ICD Coding Using BERT Pretrainingcited by 37
  24. 2020Joint Modeling of Chest Radiographs and Radiology Reports for Pulmonary Edema Assessmentcited by 37
  25. 2021GPT-3 Models are Poor Few-Shot Learners in the Biomedical Domaincited by 36
  26. 2020Learning to Ignore: Long Document Coreference with Bounded Memory Neural Networkscited by 29
  27. 2021Assessing the Impact of Automated Suggestions on Decision Making: Domain Experts Mediate Model Errors but Take Less Initiativecited by 28
  28. 2021On Generalization in Coreference Resolutioncited by 24
  29. 2022Co-training Improves Prompt-based Learning for Large Language Modelscited by 24
  30. 2022QaNER: Prompting Question Answering Models for Few-shot Named Entity Recognitioncited by 15
  31. 2020Extracting Structured Data from Physician-Patient Conversations By Predicting Noteworthy Utterancescited by 8
  32. 2022What GPT Knows About Who is Whocited by 7
  33. 2020Intimate Partner Violence and Injury Prediction From Radiology Reportscited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 2; 2023: 68; 2024: 68; 2025: 34; 2026: 34.

206 papers.

Sort:

By area:

  1. 2023Challenges and Applications of Large Language Modelscited by 251
  2. 2023Large Language Models for Generative Information Extraction: A Surveycited by 183
  3. 2023A Survey of Large Language Models for Healthcare: from Data, Technology, and Applications to Accountability and Ethicscited by 162
  4. 2024MDAgents: An Adaptive Collaboration of LLMs for Medical Decision-Makingcited by 162
  5. 2023Med-HALT: Medical Domain Hallucination Test for Large Language Modelscited by 156
  6. 2023Domain Specialization as the Key to Make Large Language Models Disruptive: A Comprehensive Surveycited by 134
  7. 2023DeID-GPT: Zero-shot Medical Text De-Identification by GPT-4cited by 128
  8. 2023A Survey of Large Language Models in Medicine: Progress, Application, and Challengecited by 123
  9. 2023Large Language Model Is Not a Good Few-shot Information Extractor, but a Good Reranker for Hard Samples!cited by 119
  10. 2023Opportunities and Challenges for ChatGPT and Large Language Models in Biomedicine and Healthcited by 118
  11. 2023Language Models Enable Simple Systems for Generating Structured Views of Heterogeneous Data Lakescited by 101
  12. 2023Do We Still Need Clinical Language Models?cited by 97
38 more
  1. 2023A Survey of Safety and Trustworthiness of Large Language Models through the Lens of Verification and Validationcited by 91
  2. 2023A Survey of GPT-3 Family Large Language Models Including ChatGPT and GPT-4cited by 81
  3. 2023Benchmarking large language models for biomedical natural language processing applications and recommendationscited by 78
  4. 2023LLMaAA: Making Large Language Models as Active Annotatorscited by 78
  5. 2023LLMs Accelerate Annotation for Medical Information Extractioncited by 77
  6. 2023CodeIE: Large Code Generation Models are Better Few-Shot Information Extractorscited by 67
  7. 2023Mental-LLM: Leveraging Large Language Models for Mental Health Prediction via Online Text Datacited by 66
  8. 2023Large AI Models in Health Informatics: Applications, Challenges, and the Futurecited by 61
  9. 2023Coupling Large Language Models with Logic Programming for Robust and General Reasoning from Textcited by 59
  10. 2023Self-Verification Improves Few-Shot Clinical Information Extractioncited by 52
  11. 2023Publicly Shareable Clinical Large Language Model Built on Synthetic Clinical Notescited by 51
  12. 2024Theoretical Analysis of Weak-to-Strong Generalizationcited by 46
  13. 2024A Toolbox for Surfacing Health Equity Harms and Biases in Large Language Modelscited by 45
  14. 2024Understanding the Dark Side of LLMs' Intrinsic Self-Correctioncited by 42
  15. 2024LLM-Ensemble: Optimal Large Language Model Ensemble Method for E-commerce Product Attribute Value Extractioncited by 39
  16. 2024Zero-Shot Clinical Trial Patient Matching with LLMscited by 35
  17. 2023CPLLM: Clinical Prediction with Large Language Modelscited by 34
  18. 2024A Survey on Medical Large Language Models: Technology, Application, Trustworthiness, and Future Directionscited by 34
  19. 2024LongHealth: A Question Answering Benchmark with Long Clinical Documentscited by 33
  20. 2023The Shaky Foundations of Clinical Foundation Models: A Survey of Large Language Models and Foundation Models for EMRscited by 29
  21. 2023Revisiting Large Language Models as Zero-shot Relation Extractorscited by 29
  22. 2024A Survey of Useful LLM Evaluationcited by 28
  23. 2024Medical Adaptation of Large Language and Vision-Language Models: Are We Making Progress?cited by 28
  24. 2023Iterative Zero-Shot LLM Prompting for Knowledge Graph Constructioncited by 25
  25. 2023AI-Generated Content (AIGC) for Various Data Modalities: A Surveycited by 25
  26. 2023BioInstruct: Instruction Tuning of Large Language Models for Biomedical Natural Language Processingcited by 25
  27. 2024General surgery vision transformer: A video pre-trained foundation model for general surgerycited by 25
  28. 2025Large Language Models are Powerful Electronic Health Record Encoderscited by 25
  29. 2024Open (Clinical) LLMs are Sensitive to Instruction Phrasingscited by 23
  30. 2023A Comprehensive Survey on Relation Extraction: Recent Advances and New Frontierscited by 21
  31. 2023ExtractGPT: Exploring the Potential of Large Language Models for Product Attribute Value Extractioncited by 20
  32. 2023Knowledge-Infused Prompting: Assessing and Advancing Clinical Text Data Generation with Large Language Modelscited by 20
  33. 2024Using LLMs for the Extraction and Normalization of Product Attribute Valuescited by 20
  34. 2023CohortGPT: An Enhanced GPT for Participant Recruitment in Clinical Studycited by 19
  35. 2023An Empirical Evaluation of Prompting Strategies for Large Language Models in Zero-Shot Clinical Natural Language Processingcited by 19
  36. 2024Guiding Clinical Reasoning with Large Language Models via Knowledge Seedscited by 19
  37. 2024Fairness and Bias in Multimodal AI: A Surveycited by 19
  38. 2023Identifying and Extracting Rare Disease Phenotypes with Large Language Modelscited by 18

50 most cited of 206 All 206 (JSON)