# The Impact of LoRA Adapters on LLMs for Clinical Text Classification Under Computational and Data Constraints

arXiv: https://arxiv.org/abs/2407.19299 · HTML: https://mlsanity.com/paper/2407.19299/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 46 references that are in the map.
- First posted on arXiv 27 Jul 2024.

## Builds on (most cited first, 22 of 22)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognition](https://mlsanity.com/paper/2205.13535/) (2022)
- [AdapterHub: A Framework for Adapting Transformers](https://mlsanity.com/paper/2007.07779/) (2020)
- [LoRA Learns Less and Forgets Less](https://mlsanity.com/paper/2405.09673/) (2024)
- [When Scaling Meets LLM Finetuning: The Effect of Data, Model and Finetuning Method](https://mlsanity.com/paper/2402.17193/) (2024)
- [Understanding Batch Normalization](https://mlsanity.com/paper/1806.02375/) (2018)
- [Training Tips for the Transformer Model](https://mlsanity.com/paper/1804.00247/) (2018)
- [Revisiting Parameter-Efficient Tuning: Are We Really There Yet?](https://mlsanity.com/paper/2202.07962/) (2022)
- [Lightweight Adapter Tuning for Multilingual Speech Translation](https://mlsanity.com/paper/2106.01463/) (2021)
- [DrBERT: A Robust Pre-trained Model in French for Biomedical and Clinical domains](https://mlsanity.com/paper/2304.00958/) (2023)
- [Scattertext: a Browser-Based Tool for Visualizing how Corpora Differ](https://mlsanity.com/paper/1703.00565/) (2017)
- [Tiny-Attention Adapter: Contexts Are More Important Than the Number of Parameters](https://mlsanity.com/paper/2211.01979/) (2022)
- [Detecting of a Patient's Condition From Clinical Narratives Using Natural Language Representation](https://mlsanity.com/paper/2104.03969/) (2021)
- [Adaptation of Autoencoder for Sparsity Reduction From Clinical Notes Representation Learning](https://mlsanity.com/paper/2209.12831/) (2022)
- [Improving Transformer Performance for French Clinical Notes Classification Using Mixture of Experts on a Limited Dataset](https://mlsanity.com/paper/2303.12892/) (2023)
- [GRN-Transformer: Enhancing Motion Artifact Detection in PICU Photoplethysmogram Signals](https://mlsanity.com/paper/2308.03722/) (2023)
- [Label Propagation Techniques for Artifact Detection in Imbalanced Classes using Photoplethysmogram Signals](https://mlsanity.com/paper/2308.08480/) (2023)
- [Transformer Meets Gated Residual Networks To Enhance Photoplethysmogram Artifact Detection Informed by Mutual Information Neural Estimation](https://mlsanity.com/paper/2405.16177/) (2024)

## Built on by (most cited first, 4 of 4)

- [Benchmarking and Adapting On-Device LLMs for Clinical Decision Support](https://mlsanity.com/paper/2601.03266/) (2025)
- [Automatic Extraction of Structured Information from Brain MRI Reports Using an Open-Weight Large Language Model](https://mlsanity.com/paper/2606.07721/) (2026)
- [The Critical Role of Model Selection in Causal Inference: A Comparative Analysis of Classification Models within the InferBERT Framework for Pharmacovigilance](https://mlsanity.com/paper/2606.17113/) (2026)
- [Tokenized Key-Gated Adapter Routing: A Secure Access Control Mechanism Against Private Data Leakage in LLMs](https://mlsanity.com/paper/2610.00309/) (2026)

## Abstract (arXiv)

Fine-tuning Large Language Models (LLMs) for clinical Natural Language Processing (NLP) poses significant challenges due to domain gap, limited data, and stringent hardware constraints. In this study, we evaluate four adapter techniques-Adapter, Lightweight, TinyAttention, and Gated Residual Network (GRN) - equivalent to Low-Rank Adaptation (LoRA), for clinical note classification under real-world, resource-constrained conditions. All experiments were conducted on a single NVIDIA Quadro P620 GPU (2 GB VRAM, 512 CUDA cores, 1.386 TFLOPS FP32), limiting batch sizes to <8 sequences and maximum sequence length to 256 tokens. Our clinical corpus comprises only 580 000 tokens, several orders of magnitude smaller than standard LLM pre-training datasets. We fine-tuned three biomedical pre-trained LLMs (CamemBERT-bio, AliBERT, DrBERT) and two lightweight Transformer models trained from scratch. Results show that 1) adapter structures provide no consistent gains when fine-tuning biomedical LLMs under these constraints, and 2) simpler Transformers, with minimal parameter counts and training times under six hours, outperform adapter-augmented LLMs, which required over 1000 GPU-hours. Among adapters, GRN achieved the best metrics (accuracy, precision, recall, F1 = 0.88). These findings demonstrate that, in low-resource clinical settings with limited data and compute, lightweight Transformers trained from scratch offer a more practical and efficient solution than large LLMs, while GRN remains a viable adapter choice when minimal adaptation is needed.