# Clinical Prompt Learning with Frozen Language Models

arXiv: https://arxiv.org/abs/2205.05535 · HTML: https://mlsanity.com/paper/2205.05535/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 36 of its 48 references that are in the map.
- First posted on arXiv 11 May 2022.

## Builds on (most cited first, 30 of 36)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [The Power of Scale for Parameter-Efficient Prompt Tuning](https://mlsanity.com/paper/2104.08691/) (2021)
- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://mlsanity.com/paper/2107.13586/) (2021)
- [BioBERT: a pre-trained biomedical language representation model for biomedical text mining](https://mlsanity.com/paper/1901.08746/) (2019)
- [Don't Stop Pretraining: Adapt Language Models to Domains and Tasks](https://mlsanity.com/paper/2004.10964/) (2020)
- [Multitask Prompted Training Enables Zero-Shot Task Generalization](https://mlsanity.com/paper/2110.08207/) (2021)
- [Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing](https://mlsanity.com/paper/2007.15779/) (2020)
- [Publicly Available Clinical BERT Embeddings](https://mlsanity.com/paper/1904.03323/) (2019)
- [Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inference](https://mlsanity.com/paper/2001.07676/) (2020)
- [Adafactor: Adaptive Learning Rates with Sublinear Memory Cost](https://mlsanity.com/paper/1804.04235/) (2018)
- [P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Tasks](https://mlsanity.com/paper/2110.07602/) (2021)
- [Annotation Artifacts in Natural Language Inference Data](https://mlsanity.com/paper/1803.02324/) (2018)
- [Multimodal Few-Shot Learning with Frozen Language Models](https://mlsanity.com/paper/2106.13884/) (2021)
- [Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model](https://mlsanity.com/paper/2201.11990/) (2022)
- [ClinicalBERT: Modeling Clinical Notes and Predicting Hospital Readmission](https://mlsanity.com/paper/1904.05342/) (2019)
- [Transfer Learning in Biomedical Natural Language Processing: An Evaluation of BERT and ELMo on Ten Benchmarking Datasets](https://mlsanity.com/paper/1906.05474/) (2019)
- [WARP: Word-level Adversarial ReProgramming](https://mlsanity.com/paper/2101.00121/) (2021)
- [Probing Neural Network Comprehension of Natural Language Arguments](https://mlsanity.com/paper/1907.07355/) (2019)
- [Knowledgeable Prompt-tuning: Incorporating Knowledge into Prompt Verbalizer for Text Classification](https://mlsanity.com/paper/2108.02035/) (2021)
- [Recall and Learn: Fine-tuning Deep Pretrained Language Models with Less Forgetting](https://mlsanity.com/paper/2004.12651/) (2020)
- [Pretrained Transformers as Universal Computation Engines](https://mlsanity.com/paper/2103.05247/) (2021)
- [OpenPrompt: An Open-source Framework for Prompt-learning](https://mlsanity.com/paper/2111.01998/) (2021)
- [MIMIC-Extract: A Data Extraction, Preprocessing, and Representation Pipeline for MIMIC-III](https://mlsanity.com/paper/1907.08322/) (2019)
- [Code Synonyms Do Matter: Multiple Synonyms Matching Network for Automatic ICD Coding](https://mlsanity.com/paper/2203.01515/) (2022)
- [Clinical Outcome Prediction from Admission Notes using Self-Supervised Knowledge Integration](https://mlsanity.com/paper/2102.04110/) (2021)
- [GPT-3 Models are Poor Few-Shot Learners in the Biomedical Domain](https://mlsanity.com/paper/2109.02555/) (2021)

## Built on by (most cited first, 19 of 19)

- [Open-Ended Medical Visual Question Answering Through Prefix Tuning of Language Models](https://mlsanity.com/paper/2303.05977/) (2023)
- [Prompt Engineering for Healthcare: Methodologies and Applications](https://mlsanity.com/paper/2304.14670/) (2023)
- [Evaluating Large Language Models on a Highly-specialized Topic, Radiation Oncology Physics](https://mlsanity.com/paper/2304.01938/) (2023)
- [Knowledge Injected Prompt Based Fine-tuning for Multi-label Few-shot ICD Coding](https://mlsanity.com/paper/2210.03304/) (2022)
- [Few-shot Adaptation of Medical Vision-Language Models](https://mlsanity.com/paper/2409.03868/) (2024)
- [Identifying and Extracting Rare Disease Phenotypes with Large Language Models](https://mlsanity.com/paper/2306.12656/) (2023)
- [Multi-label Few-shot ICD Coding as Autoregressive Generation with Prompt](https://mlsanity.com/paper/2211.13813/) (2022)
- [Prompt engineering paradigms for medical applications: scoping review and recommendations for better practices](https://mlsanity.com/paper/2405.01249/) (2024)
- [Exploiting prompt learning with pre-trained language models for Alzheimer's Disease detection](https://mlsanity.com/paper/2210.16539/) (2022)
- [Surpassing GPT-4 Medical Coding with a Two-Stage Approach](https://mlsanity.com/paper/2311.13735/) (2023)
- [Efficiency at Scale: Investigating the Performance of Diminutive Language Models in Clinical Tasks](https://mlsanity.com/paper/2402.10597/) (2024)
- [ChronoLLM: Customizing Language Models for Physics-Based Simulation Code Generation](https://mlsanity.com/paper/2508.13975/) (2025)
- [Leveraging GPT-4 for Food Effect Summarization to Enhance Product-Specific Guidance Development via Iterative Prompting](https://mlsanity.com/paper/2306.16275/) (2023)
- [Detecting the Clinical Features of Difficult-to-Treat Depression using Synthetic Data from Large Language Models](https://mlsanity.com/paper/2402.07645/) (2024)
- [Clinical information extraction for Low-resource languages with Few-shot learning using Pre-trained language models and Prompting](https://mlsanity.com/paper/2403.13369/) (2024)
- [Bespoke Large Language Models for Digital Triage Assistance in Mental Health Care](https://mlsanity.com/paper/2403.19790/) (2024)
- [Developing Healthcare Language Model Embedding Spaces](https://mlsanity.com/paper/2403.19802/) (2024)
- [Eye-SpatialNet: Spatial Information Extraction from Ophthalmology Notes](https://mlsanity.com/paper/2305.11948/) (2023)
- [Large Language Models Perform on Par with Experts Identifying Mental Health Factors in Adolescent Online Forums](https://mlsanity.com/paper/2404.16461/) (2024)

## Abstract (arXiv)

Prompt learning is a new paradigm in the Natural Language Processing (NLP) field which has shown impressive performance on a number of natural language tasks with common benchmarking text datasets in full, few-shot, and zero-shot train-evaluation setups. Recently, it has even been observed that large but frozen pre-trained language models (PLMs) with prompt learning outperform smaller but fine-tuned models. However, as with many recent NLP trends, the performance of even the largest PLMs such as GPT-3 do not perform well on specialized domains (e.g. medical text), and the common practice to achieve State of the Art (SoTA) results still consists of pre-training and fine-tuning the PLMs on downstream tasks. The reliance on fine-tuning large PLMs is problematic in clinical settings where data is often held in non-GPU environments, and more resource efficient methods of training specialized domain models is crucial. We investigated the viability of prompt learning on clinically meaningful decision tasks and directly compared with more traditional fine-tuning methods. Results are partially in line with the prompt learning literature, with prompt learning able to match or improve on traditional fine-tuning with substantially fewer trainable parameters and requiring less training data. We argue that prompt learning therefore provides lower computational resource costs applicable to clinical settings, that can serve as an alternative to fine-tuning ever increasing in size PLMs. Complementary code to reproduce experiments presented in this work can be found at: https://github.com/NtaylorOX/Public_Clinical_Prompt.