mlsanityData to 7 Oct 2026

Parameter-Efficient Fine-Tuning of LLaMA for the Clinical Domain

arXiv 2307.03042 ↗First posted 6 Jul 2023cs.CL cs.LGcited by 38

Not on the map. Why?

Abstract (arXiv)

Adapting pretrained language models to novel domains, such as clinical applications, traditionally involves retraining their entire set of parameters. Parameter-Efficient Fine-Tuning (PEFT) techniques for fine-tuning language models significantly reduce computational requirements by selectively fine-tuning small subsets of parameters. In this study, we propose a two-step PEFT framework and evaluate it in the clinical domain. Our approach combines a specialised PEFT adapter layer designed for clinical domain adaptation with another adapter specialised for downstream tasks. We evaluate the framework on multiple clinical outcome prediction datasets, comparing it to clinically trained language models. Our framework achieves a better AUROC score averaged across all clinical downstream tasks compared to clinical language models. In particular, we observe large improvements of 4-5% AUROC in large-scale multilabel classification tasks, such as diagnoses and procedures classification. To our knowledge, this study is the first to provide an extensive empirical analysis of the interplay between PEFT techniques and domain adaptation in an important real-world domain of clinical applications.

Key facts. Cited by 38 papers in the map (data through 7 Oct 2026). Builds on 29 of its 40 references that are in the map. First posted on arXiv 6 Jul 2023.

Builds on

29 of 40 references.

References · 29

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  6. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  7. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  8. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  9. 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,935
  10. 2022Large Language Models Encode Clinical Knowledgecited by 1,807
  11. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,548
  12. 2019Publicly Available Clinical BERT Embeddingscited by 995
17 more
  1. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 776
  2. 2018A Tutorial on Bayesian Optimizationcited by 734
  3. 2021GPT Understands, Toocited by 715
  4. 2021P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Taskscited by 686
  5. 2022BioGPT: Generative Pre-trained Transformer for Biomedical Text Generation and Miningcited by 531
  6. 2023Towards Expert-Level Medical Question Answering with Large Language Modelscited by 382
  7. 2019Transfer Learning in Biomedical Natural Language Processing: An Evaluation of BERT and ELMo on Ten Benchmarking Datasetscited by 298
  8. 2022GatorTron: A Large Clinical Language Model to Unlock Patient Information from Unstructured Electronic Health Recordscited by 238
  9. 2023Do We Still Need Clinical Language Models?cited by 97
  10. 2022Clinical-Longformer and Clinical-BigBird: Transformers for long clinical sequencescited by 67
  11. 2020UmlsBERT: Clinical Domain Knowledge Augmentation of Contextual Embeddings Using the Unified Medical Language System Metathesauruscited by 53
  12. 2021Clinical Outcome Prediction from Admission Notes using Self-Supervised Knowledge Integrationcited by 43
  13. 2020Rethinking Distributional Matching Based Domain Adaptationcited by 33
  14. 2019Clinical XLNet: Modeling Sequential Clinical Notes and Predicting Prolonged Mechanical Ventilationcited by 30
  15. 2021Literature-Augmented Clinical Outcome Predictioncited by 20
  16. 2021Unsupervised Domain Adaptation with Adaptercited by 14
  17. 2023UDApter -- Efficient Domain Adaptation Using Adapterscited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 20; 2025: 14; 2026: 3.

38 papers.

Sort:

By area:

  1. 2023LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA Compositioncited by 359
  2. 2024A Survey on LoRA of Large Language Modelscited by 67
  3. 2024A Comprehensive Survey of Large Language Models and Multimodal Large Language Models in Medicinecited by 45
  4. 2024MedAdapter: Efficient Test-Time Adaptation of Large Language Models towards Medical Reasoningcited by 35
  5. 2024JMLR: Joint Medical LLM and Retrieval Training for Enhancing Reasoning and Professional Question Answering Capabilitycited by 32
  6. 2025Llama-3.1-FoundationAI-SecurityLLM-Base-8B Technical Reportcited by 28
  7. 2024Data-Centric Foundation Models in Computational Healthcare: A Surveycited by 27
  8. 2024Query Performance Prediction using Relevance Judgments Generated by Large Language Modelscited by 18
  9. 2025Aligned but Blind: Alignment Increases Implicit Bias by Reducing Awareness of Racecited by 15
  10. 2024Me LLaMA: Foundation Large Language Models for Medical Applicationscited by 10
  11. 2024RoQLlama: A Lightweight Romanian Adapted Language Modelcited by 7
  12. 2024Fusion of Domain-Adapted Vision and Language Models for Medical Visual Question Answeringcited by 5
26 more
  1. 2024Worldwide Federated Training of Language Modelscited by 5
  2. 2024Efficient Language Adaptive Pre-training: Extending State-of-the-Art Large Language Models for Polishcited by 4
  3. 2024Efficiency at Scale: Investigating the Performance of Diminutive Language Models in Clinical Taskscited by 4
  4. 2024DeLLiriuM: A large language model for delirium prediction in the ICU using structured EHRcited by 4
  5. 2025Evolution of meta's llama models and parameter-efficient fine-tuning of large language models: a surveycited by 4
  6. 2024Are Clinical T5 Models Better for Clinical Text?cited by 3
  7. 2025Dialogue is Better Than Monologue: Instructing Medical LLMs via Strategical Conversationscited by 3
  8. 2025Benchmarking Foundation Models and Parameter-Efficient Fine-Tuning for Prognosis Prediction in Medical Imagingcited by 3
  9. 2024Edinburgh Clinical NLP at SemEval-2024 Task 2: Fine-tune your model unless you have access to GPT-4cited by 2
  10. 2024Edinburgh Clinical NLP at MEDIQA-CORR 2024: Guiding Large Language Models with Hintscited by 2
  11. 2025ELMTEX: Fine-Tuning Large Language Models for Structured Clinical Information Extraction. A Case Study on Clinical Reportscited by 2
  12. 2024The Right Model for the Job: An Evaluation of Legal Multi-Label Classification Baselinescited by 1
  13. 2024Assertion Detection Large Language Model In-context Learning LoRA Fine-tuningcited by 1
  14. 2025QUAD-LLM-MLTC: Large Language Models Ensemble Learning for Healthcare Text Multi-Label Classificationcited by 1
  15. 2025MSPLoRA: A Multi-Scale Pyramid Low-Rank Adaptation for Efficient Model Fine-Tuningcited by 1
  16. 2025Aligning Large Language Models with Healthcare Stakeholders: A Pathway to Trustworthy AI Integrationcited by 1
  17. 2025CliniBench: A Clinical Outcome Prediction Benchmark for Generative and Encoder-Based Language Modelscited by 1
  18. 2025OncoReason: Structuring Clinical Reasoning in LLMs for Robust and Interpretable Survival Predictioncited by 1
  19. 2024A Comparative Study on Patient Language across Therapeutic Domains for Effective Patient Voice Classification in Online Health Discussionscited by 0
  20. 2024Towards Efficient Large Language Models for Scientific Text: A Reviewcited by 0
  21. 2025What do Large Language Models know about materials?cited by 0
  22. 2025HapticLLaMA: A Multimodal Sensory Language Model for Haptic Captioningcited by 0
  23. 2025Building Patient Journeys in Hebrew: A Language Model for Clinical Timeline Extractioncited by 0
  24. 2026Trade-offs in Medical LLM Adaptation: An Empirical Study in French QAcited by 0
  25. 2026Locally Deployable Small Language Models for Emergency Department Decision Support: A Systematic Benchmark of Fine-Tuning Strategiescited by 0
  26. 2026SynEHR: Joint Modeling Inter-visit Temporal Evolution and Intra-visit Clinical Structure for Longitudinal EHR Synthesiscited by 0