# Model Tuning or Prompt Tuning? A Study of Large Language Models for Clinical Concept and Relation Extraction

arXiv: https://arxiv.org/abs/2310.06239 · HTML: https://mlsanity.com/paper/2310.06239/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 56 references that are in the map.
- First posted on arXiv 10 Oct 2023.

## Builds on (most cited first, 23 of 23)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [The Power of Scale for Parameter-Efficient Prompt Tuning](https://mlsanity.com/paper/2104.08691/) (2021)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://mlsanity.com/paper/2107.13586/) (2021)
- [Language Models as Knowledge Bases?](https://mlsanity.com/paper/1909.01066/) (2019)
- [How Can We Know What Language Models Know?](https://mlsanity.com/paper/1911.12543/) (2019)
- [GPT Understands, Too](https://mlsanity.com/paper/2103.10385/) (2021)
- [P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Tasks](https://mlsanity.com/paper/2110.07602/) (2021)
- [Learning How to Ask: Querying LMs with Mixtures of Soft Prompts](https://mlsanity.com/paper/2104.06599/) (2021)
- [GatorTron: A Large Clinical Language Model to Unlock Patient Information from Unstructured Electronic Health Records](https://mlsanity.com/paper/2203.03540/) (2022)
- [PPT: Pre-trained Prompt Tuning for Few-shot Learning](https://mlsanity.com/paper/2109.04332/) (2021)
- [A Unified MRC Framework for Named Entity Recognition](https://mlsanity.com/paper/1910.11476/) (2019)
- [Entity-Relation Extraction as Multi-Turn Question Answering](https://mlsanity.com/paper/1905.05529/) (2019)
- [Clinical-Longformer and Clinical-BigBird: Transformers for long clinical sequences](https://mlsanity.com/paper/2201.11838/) (2022)
- [The 2022 n2c2/UW Shared Task on Extracting Social Determinants of Health](https://mlsanity.com/paper/2301.05571/) (2023)
- [Clinical Concept and Relation Extraction Using Prompt-based Machine Reading Comprehension](https://mlsanity.com/paper/2303.08262/) (2023)
- [Clinical Relation Extraction Using Transformer-based Models](https://mlsanity.com/paper/2107.08957/) (2021)
- [Recurrent neural networks with specialized word embeddings for health-domain named-entity recognition](https://mlsanity.com/paper/1706.09569/) (2017)

## Built on by (most cited first, 30 of 32)

- [Exploring Autonomous Agents through the Lens of Large Language Models: A Review](https://mlsanity.com/paper/2404.04442/) (2024)
- [TWIN-GPT: Digital Twins for Clinical Trials via Large Language Model](https://mlsanity.com/paper/2404.01273/) (2024)
- [Prompt engineering paradigms for medical applications: scoping review and recommendations for better practices](https://mlsanity.com/paper/2405.01249/) (2024)
- [Empowering Healthcare Practitioners with Language Models: Structuring Speech Transcripts in Two Real-World Clinical Applications](https://mlsanity.com/paper/2507.05517/) (2025)
- [Generative Large Language Models Are All-purpose Text Analytics Engines: Text-to-text Learning Is All Your Need](https://mlsanity.com/paper/2312.06099/) (2023)
- [Automatic Summarization of Doctor-Patient Encounter Dialogues Using Large Language Model through Prompt Tuning](https://mlsanity.com/paper/2403.13089/) (2024)
- [Leveraging Large Language Models to Extract Information on Substance Use Disorder Severity from Clinical Notes: A Zero-shot Learning Approach](https://mlsanity.com/paper/2403.12297/) (2024)
- [Automating PTSD Diagnostics in Clinical Interviews: Leveraging Large Language Models for Trauma Assessments](https://mlsanity.com/paper/2405.11178/) (2024)
- [An LLM-Powered Agent for Physiological Data Analysis: A Case Study on PPG-based Heart Rate Estimation](https://mlsanity.com/paper/2502.12836/) (2025)
- [Narrative Feature or Structured Feature? A Study of Large Language Models to Identify Cancer Patients at Risk of Heart Failure](https://mlsanity.com/paper/2403.11425/) (2024)
- [Structured Gradient Guidance for Few-Shot Adaptation in Large Language Models](https://mlsanity.com/paper/2506.00726/) (2025)
- [Grounded Knowledge-Enhanced Medical Vision-Language Pre-training for Chest X-Ray](https://mlsanity.com/paper/2404.14750/) (2024)
- [ELMTEX: Fine-Tuning Large Language Models for Structured Clinical Information Extraction. A Case Study on Clinical Reports](https://mlsanity.com/paper/2502.05638/) (2025)
- [Rumor Detection by Multi-task Suffix Learning based on Time-series Dual Sentiments](https://mlsanity.com/paper/2502.14383/) (2025)
- [Graph Your Own Prompt](https://mlsanity.com/paper/2509.23373/) (2025)
- [Overview of the MEDIQA-OE 2025 Shared Task on Medical Order Extraction from Doctor-Patient Consultations](https://mlsanity.com/paper/2510.26974/) (2025)
- [Enhancing Biomedical Relation Extraction with Directionality](https://mlsanity.com/paper/2501.14079/) (2025)
- [MuTRAP: Multi-trigger Trojans Attacking Robot Task Planning Systems](https://mlsanity.com/paper/2504.17070/) (2025)
- [A Study of Large Language Models for Patient Information Extraction: Model Architecture, Fine-Tuning Strategy, and Multi-task Instruction Tuning](https://mlsanity.com/paper/2509.04753/) (2025)
- [Improving Generalizability of Extracting Social Determinants of Health Using Large Language Models through Prompt-tuning](https://mlsanity.com/paper/2403.12374/) (2024)
- [Towards Efficient Large Language Models for Scientific Text: A Review](https://mlsanity.com/paper/2408.10729/) (2024)
- [Assistive AI for Augmenting Human Decision-making](https://mlsanity.com/paper/2410.14353/) (2024)
- [Advancing Pancreatic Cancer Prediction with a Next Visit Token Prediction Head on top of Med-BERT](https://mlsanity.com/paper/2501.02044/) (2025)
- [Initial Investigation of LLM-Assisted Development of Rule-Based Clinical NLP System](https://mlsanity.com/paper/2506.16628/) (2025)
- [Dual LoRA: Enhancing LoRA with Magnitude and Direction Updates](https://mlsanity.com/paper/2512.03402/) (2025)
- [Improving Automatic Summarization of Radiology Reports through Mid-Training of Large Language Models](https://mlsanity.com/paper/2603.19275/) (2026)
- [An Initial Exploration of Contrastive Prompt Tuning to Generate Energy-Efficient Code](https://mlsanity.com/paper/2604.02352/) (2026)
- [OMEGA: Optimizing Machine Learning by Evaluating Generated Algorithms](https://mlsanity.com/paper/2604.26211/) (2026)
- [Learning to Translate from Soft to Hard LLM Prompts](https://mlsanity.com/paper/2605.27642/) (2026)
- [Few-Shot Biomedical Relation Extraction with Large Language Models: A Viable Alternative to Supervised Learning?](https://mlsanity.com/paper/2606.15412/) (2026)

## Abstract (arXiv)

Objective To develop soft prompt-based learning algorithms for large language models (LLMs), examine the shape of prompts, prompt-tuning using frozen/unfrozen LLMs, transfer learning, and few-shot learning abilities. Methods We developed a soft prompt-based LLM model and compared 4 training strategies including (1) fine-tuning without prompts; (2) hard-prompt with unfrozen LLMs; (3) soft-prompt with unfrozen LLMs; and (4) soft-prompt with frozen LLMs. We evaluated 7 pretrained LLMs using the 4 training strategies for clinical concept and relation extraction on two benchmark datasets. We evaluated the transfer learning ability of the prompt-based learning algorithms in a cross-institution setting. We also assessed the few-shot learning ability. Results and Conclusion When LLMs are unfrozen, GatorTron-3.9B with soft prompting achieves the best strict F1-scores of 0.9118 and 0.8604 for concept extraction, outperforming the traditional fine-tuning and hard prompt-based models by 0.6~3.1% and 1.2~2.9%, respectively; GatorTron-345M with soft prompting achieves the best F1-scores of 0.8332 and 0.7488 for end-to-end relation extraction, outperforming the other two models by 0.2~2% and 0.6~11.7%, respectively. When LLMs are frozen, small (i.e., 345 million parameters) LLMs have a big gap to be competitive with unfrozen models; scaling LLMs up to billions of parameters makes frozen LLMs competitive with unfrozen LLMs. For cross-institute evaluation, soft prompting with a frozen GatorTron-8.9B model achieved the best performance. This study demonstrates that (1) machines can learn soft prompts better than humans, (2) frozen LLMs have better few-shot learning ability and transfer learning ability to facilitate muti-institution applications, and (3) frozen LLMs require large models.