# Prompt Combines Paraphrase: Teaching Pre-trained Models to Understand Rare Biomedical Words

arXiv: https://arxiv.org/abs/2209.06453 · HTML: https://mlsanity.com/paper/2209.06453/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 37 references that are in the map.
- First posted on arXiv 14 Sep 2022.

## Builds on (most cited first, 30 of 30)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://mlsanity.com/paper/2107.13586/) (2021)
- [BioBERT: a pre-trained biomedical language representation model for biomedical text mining](https://mlsanity.com/paper/1901.08746/) (2019)
- [SciBERT: A Pretrained Language Model for Scientific Text](https://mlsanity.com/paper/1903.10676/) (2019)
- [Making Pre-trained Language Models Better Few-shot Learners](https://mlsanity.com/paper/2012.15723/) (2020)
- [Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing](https://mlsanity.com/paper/2007.15779/) (2020)
- [Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inference](https://mlsanity.com/paper/2001.07676/) (2020)
- [Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference](https://mlsanity.com/paper/1902.01007/) (2019)
- [P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Tasks](https://mlsanity.com/paper/2110.07602/) (2021)
- [Pre-trained Models for Natural Language Processing: A Survey](https://mlsanity.com/paper/2003.08271/) (2020)
- [ClinicalBERT: Modeling Clinical Notes and Predicting Hospital Readmission](https://mlsanity.com/paper/1904.05342/) (2019)
- [Transfer Learning in Biomedical Natural Language Processing: An Evaluation of BERT and ELMo on Ten Benchmarking Datasets](https://mlsanity.com/paper/1906.05474/) (2019)
- [Representation Degeneration Problem in Training Natural Language Generation Models](https://mlsanity.com/paper/1907.12009/) (2019)
- [AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts](https://mlsanity.com/paper/2010.15980/) (2020)
- [PPT: Pre-trained Prompt Tuning for Few-shot Learning](https://mlsanity.com/paper/2109.04332/) (2021)
- [ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfer](https://mlsanity.com/paper/2105.11741/) (2021)
- [Lessons from Natural Language Inference in the Clinical Domain](https://mlsanity.com/paper/1808.06752/) (2018)
- [BioMegatron: Larger Biomedical Domain Language Model](https://mlsanity.com/paper/2010.06060/) (2020)
- [UmlsBERT: Clinical Domain Knowledge Augmentation of Contextual Embeddings Using the Unified Medical Language System Metathesaurus](https://mlsanity.com/paper/2010.10391/) (2020)
- [Rare Words: A Major Problem for Contextualized Embeddings And How to Fix it by Attentive Mimicking](https://mlsanity.com/paper/1904.06707/) (2019)
- [MedSTS: A Resource for Clinical Semantic Textual Similarity](https://mlsanity.com/paper/1808.09397/) (2018)
- [Dict-BERT: Enhancing Language Model Pre-training with Dictionary](https://mlsanity.com/paper/2110.06490/) (2021)
- [SMedBERT: A Knowledge-Enhanced Pre-trained Language Model with Structured Semantics for Medical Text Mining](https://mlsanity.com/paper/2108.08983/) (2021)
- [A Short Review of Ethical Challenges in Clinical Natural Language Processing](https://mlsanity.com/paper/1703.10090/) (2017)
- [Taking Notes on the Fly Helps BERT Pre-training](https://mlsanity.com/paper/2008.01466/) (2020)

## Built on by (most cited first, 6 of 6)

- [Prompt Engineering for Healthcare: Methodologies and Applications](https://mlsanity.com/paper/2304.14670/) (2023)
- [Knowledge-tuning Large Language Models with Structured Medical Knowledge Bases for Reliable Response Generation in Chinese](https://mlsanity.com/paper/2309.04175/) (2023)
- [Improving Biomedical Entity Linking with Retrieval-enhanced Learning](https://mlsanity.com/paper/2312.09806/) (2023)
- [Manifold-based Verbalizer Space Re-embedding for Tuning-free Prompt-based Classification](https://mlsanity.com/paper/2309.04174/) (2023)
- [Iterative Prompt Refinement for Radiation Oncology Symptom Extraction Using Teacher-Student Large Language Models](https://mlsanity.com/paper/2402.04075/) (2024)
- [Modeling Data Diversity for Joint Instance and Verbalizer Selection in Cold-Start Scenarios](https://mlsanity.com/paper/2507.00330/) (2025)

## Abstract (arXiv)

Prompt-based fine-tuning for pre-trained models has proven effective for many natural language processing tasks under few-shot settings in general domain. However, tuning with prompt in biomedical domain has not been investigated thoroughly. Biomedical words are often rare in general domain, but quite ubiquitous in biomedical contexts, which dramatically deteriorates the performance of pre-trained models on downstream biomedical applications even after fine-tuning, especially in low-resource scenarios. We propose a simple yet effective approach to helping models learn rare biomedical words during tuning with prompt. Experimental results show that our method can achieve up to 6% improvement in biomedical natural language inference task without any extra parameters or training steps using few-shot vanilla prompt settings.