# An In-Depth Evaluation of Federated Learning on Biomedical Natural Language Processing

arXiv: https://arxiv.org/abs/2307.11254 · HTML: https://mlsanity.com/paper/2307.11254/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 43 references that are in the map.
- First posted on arXiv 20 Jul 2023.

## Builds on (most cited first, 24 of 24)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [BioBERT: a pre-trained biomedical language representation model for biomedical text mining](https://mlsanity.com/paper/1901.08746/) (2019)
- [Federated Learning: Strategies for Improving Communication Efficiency](https://mlsanity.com/paper/1610.05492/) (2016)
- [PaLM 2 Technical Report](https://mlsanity.com/paper/2305.10403/) (2023)
- [Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing](https://mlsanity.com/paper/2007.15779/) (2020)
- [Publicly Available Clinical BERT Embeddings](https://mlsanity.com/paper/1904.03323/) (2019)
- [Plug and Play Language Models: A Simple Approach to Controlled Text Generation](https://mlsanity.com/paper/1912.02164/) (2019)
- [Bidirectional LSTM-CRF Models for Sequence Tagging](https://mlsanity.com/paper/1508.01991/) (2015)
- [Federated Learning with Differential Privacy: Algorithms and Performance Analysis](https://mlsanity.com/paper/1911.00222/) (2019)
- [How to Fine-Tune BERT for Text Classification?](https://mlsanity.com/paper/1905.05583/) (2019)
- [BERT Post-Training for Review Reading Comprehension and Aspect-based Sentiment Analysis](https://mlsanity.com/paper/1904.02232/) (2019)
- [Simple BERT Models for Relation Extraction and Semantic Role Labeling](https://mlsanity.com/paper/1904.05255/) (2019)
- [End-to-End Open-Domain Question Answering with BERTserini](https://mlsanity.com/paper/1902.01718/) (2019)
- [Federated Learning for Open Banking](https://mlsanity.com/paper/2108.10749/) (2021)
- [FedNLP: Benchmarking Federated Learning Methods for Natural Language Processing Tasks](https://mlsanity.com/paper/2104.08815/) (2021)
- [Benchmarking large language models for biomedical natural language processing applications and recommendations](https://mlsanity.com/paper/2305.16326/) (2023)
- [BERT with History Answer Embedding for Conversational Question Answering](https://mlsanity.com/paper/1905.05412/) (2019)
- [Fine-Tuning Large Neural Language Models for Biomedical Natural Language Processing](https://mlsanity.com/paper/2112.07869/) (2021)
- [Deep Federated Learning for Autonomous Driving](https://mlsanity.com/paper/2110.05754/) (2021)
- [Federated Learning Meets Natural Language Processing: A Survey](https://mlsanity.com/paper/2107.12603/) (2021)
- [Federated pretraining and fine tuning of BERT using clinical notes from multiple silos](https://mlsanity.com/paper/2002.08562/) (2020)

## Built on by (most cited first, 1 of 1)

- [Federated Learning with Layer Skipping: Efficient Training of Large Language Models for Healthcare NLP](https://mlsanity.com/paper/2504.10536/) (2025)

## Abstract (arXiv)

Language models (LMs) such as BERT and GPT have revolutionized natural language processing (NLP). However, the medical field faces challenges in training LMs due to limited data access and privacy constraints imposed by regulations like the Health Insurance Portability and Accountability Act (HIPPA) and the General Data Protection Regulation (GDPR). Federated learning (FL) offers a decentralized solution that enables collaborative learning while ensuring data privacy. In this study, we evaluated FL on 2 biomedical NLP tasks encompassing 8 corpora using 6 LMs. Our results show that: 1) FL models consistently outperformed models trained on individual clients' data and sometimes performed comparably with models trained with polled data; 2) with the fixed number of total data, FL models training with more clients produced inferior performance but pre-trained transformer-based models exhibited great resilience. 3) FL models significantly outperformed large language models using zero-/one-shot learning and offered lightning inference speed.