# Federated Learning with Layer Skipping: Efficient Training of Large Language Models for Healthcare NLP

arXiv: https://arxiv.org/abs/2504.10536 · HTML: https://mlsanity.com/paper/2504.10536/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 16 references that are in the map.
- First posted on arXiv 13 Apr 2025.

## Builds on (most cited first, 9 of 9)

- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [Deep Learning with Differential Privacy](https://mlsanity.com/paper/1607.00133/) (2016)
- [Federated Learning with Personalization Layers](https://mlsanity.com/paper/1912.00818/) (2019)
- [Split learning for health: Distributed deep learning without sharing raw patient data](https://mlsanity.com/paper/1812.00564/) (2018)
- [Recent Methodological Advances in Federated Learning for Healthcare](https://mlsanity.com/paper/2310.02874/) (2023)
- [An In-Depth Evaluation of Federated Learning on Biomedical Natural Language Processing](https://mlsanity.com/paper/2307.11254/) (2023)

## Built on by (most cited first, 2 of 2)

- [LBM-GNN: Graph Neural Network Enhanced Lattice Boltzmann Method](https://mlsanity.com/paper/2504.14494/) (2025)
- [TIGER: Inverting Transformer Gradients via Embedding-Subspace Distance Optimization](https://mlsanity.com/paper/2606.18312/) (2026)

## Abstract (arXiv)

Federated learning (FL) enables collaborative model training across organizations without sharing raw data, addressing crucial privacy concerns in healthcare natural language processing (NLP). However, training large language models (LLMs) in federated settings faces significant challenges, including communication overhead and data heterogeneity. We propose Layer-Skipping Federated Learning, where only selected layers of a pre-trained LLM are fine-tuned across clients while others remain frozen. Applied to LLaMA 3.2-1B, our approach reduces communication costs by approximately 70% while maintaining performance within 2% of centralized training. We evaluate our method on clinical NER and classification tasks using i2b2 and MIMIC-III datasets. Our experiments demonstrate that Layer-Skipping FL outperforms competitive baselines, handles non-IID clinical data distributions effectively, and shows robustness when combined with differential privacy. This approach represents a practical solution for privacy-preserving collaborative learning in healthcare NLP.