# Exploring the Effectiveness of Instruction Tuning in Biomedical Language Processing

arXiv: https://arxiv.org/abs/2401.00579 · HTML: https://mlsanity.com/paper/2401.00579/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 28 references that are in the map.
- First posted on arXiv 31 Dec 2023.

## Builds on (most cited first, 16 of 16)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Exams](https://mlsanity.com/paper/2009.13081/) (2020)
- [PubMedQA: A Dataset for Biomedical Research Question Answering](https://mlsanity.com/paper/1909.06146/) (2019)
- [The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only](https://mlsanity.com/paper/2306.01116/) (2023)
- [ChatDoctor: A Medical Chat Model Fine-Tuned on a Large Language Model Meta-AI (LLaMA) Using Medical Domain Knowledge](https://mlsanity.com/paper/2303.14070/) (2023)
- [MedAlpaca -- An Open-Source Collection of Medical Conversational AI Models and Training Data](https://mlsanity.com/paper/2304.08247/) (2023)
- [Lessons from Natural Language Inference in the Clinical Domain](https://mlsanity.com/paper/1808.06752/) (2018)
- [Do We Still Need Clinical Language Models?](https://mlsanity.com/paper/2302.08091/) (2023)
- [A Comprehensive Evaluation of Large Language Models on Benchmark Biomedical Text Processing Tasks](https://mlsanity.com/paper/2310.04270/) (2023)
- [Med7: a transferable clinical natural language processing model for electronic health records](https://mlsanity.com/paper/2003.01271/) (2020)
- [On the Effectiveness of Compact Biomedical Transformers](https://mlsanity.com/paper/2209.03182/) (2022)

## Built on by (most cited first, 16 of 16)

- [SciRIFF: A Resource to Enhance Language Model Instruction-Following over Scientific Literature](https://mlsanity.com/paper/2406.07835/) (2024)
- [TIMER: Temporal Instruction Modeling and Evaluation for Longitudinal Clinical Records](https://mlsanity.com/paper/2503.04176/) (2025)
- [LLMs in Biomedicine: A study on clinical Named Entity Recognition](https://mlsanity.com/paper/2404.07376/) (2024)
- [Agentic Medical Knowledge Graphs Enhance Medical Question Answering: Bridging the Gap Between LLMs and Evolving Medical Knowledge](https://mlsanity.com/paper/2502.13010/) (2025)
- [Large Language Models Struggle in Token-Level Clinical Named Entity Recognition](https://mlsanity.com/paper/2407.00731/) (2024)
- [Do LLMs Surpass Encoders for Biomedical NER?](https://mlsanity.com/paper/2504.00664/) (2025)
- [Efficiency at Scale: Investigating the Performance of Diminutive Language Models in Clinical Tasks](https://mlsanity.com/paper/2402.10597/) (2024)
- [BioMistral-NLU: Towards More Generalizable Medical Language Understanding through Instruction Tuning](https://mlsanity.com/paper/2410.18955/) (2024)
- [Temporal Relation Extraction in Clinical Texts: A Span-based Graph Transformer Approach](https://mlsanity.com/paper/2503.18085/) (2025)
- [Investigating Task Arithmetic for Zero-Shot Information Retrieval](https://mlsanity.com/paper/2505.00649/) (2025)
- [Natural Language Processing in Support of Evidence-based Medicine: A Scoping Review](https://mlsanity.com/paper/2505.22280/) (2025)
- [Assessing biomedical knowledge robustness in large language models by query-efficient sampling attacks](https://mlsanity.com/paper/2402.10527/) (2024)
- [Unraveling the Dominance of Large Language Models Over Transformer Models for Bangla Natural Language Inference: A Comprehensive Study](https://mlsanity.com/paper/2405.02937/) (2024)
- [Rapid Biomedical Research Classification: The Pandemic PACT Advanced Categorisation Engine](https://mlsanity.com/paper/2407.10086/) (2024)
- [Towards Scalable and Cross-Lingual Specialist Language Models for Oncology](https://mlsanity.com/paper/2503.08323/) (2025)
- [FMI@SU ToxHabits: Evaluating LLMs Performance on Toxic Habit Extraction in Spanish Clinical Texts](https://mlsanity.com/paper/2604.06403/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs), particularly those similar to ChatGPT, have significantly influenced the field of Natural Language Processing (NLP). While these models excel in general language tasks, their performance in domain-specific downstream tasks such as biomedical and clinical Named Entity Recognition (NER), Relation Extraction (RE), and Medical Natural Language Inference (NLI) is still evolving. In this context, our study investigates the potential of instruction tuning for biomedical language processing, applying this technique to two general LLMs of substantial scale. We present a comprehensive, instruction-based model trained on a dataset that consists of approximately 200,000 instruction-focused samples. This dataset represents a carefully curated compilation of existing data, meticulously adapted and reformatted to align with the specific requirements of our instruction-based tasks. This initiative represents an important step in utilising such models to achieve results on par with specialised encoder-only models like BioBERT and BioClinicalBERT for various classical biomedical NLP tasks. Our work includes an analysis of the dataset's composition and its impact on model performance, providing insights into the intricacies of instruction tuning. By sharing our codes, models, and the distinctively assembled instruction-based dataset, we seek to encourage ongoing research and development in this area.