# Federated pretraining and fine tuning of BERT using clinical notes from multiple silos

arXiv: https://arxiv.org/abs/2002.08562 · HTML: https://mlsanity.com/paper/2002.08562/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Feb 2020.

## Built on by (most cited first, 18 of 18)

- [FedNLP: Benchmarking Federated Learning Methods for Natural Language Processing Tasks](https://mlsanity.com/paper/2104.08815/) (2021)
- [Recovering Private Text in Federated Learning of Language Models](https://mlsanity.com/paper/2205.08514/) (2022)
- [Federated Learning Meets Natural Language Processing: A Survey](https://mlsanity.com/paper/2107.12603/) (2021)
- [Scaling Federated Learning for Fine-tuning of Large Language Models](https://mlsanity.com/paper/2102.00875/) (2021)
- [Pretrained Models for Multilingual Federated Learning](https://mlsanity.com/paper/2206.02291/) (2022)
- [Scaling Language Model Size in Cross-Device Federated Learning](https://mlsanity.com/paper/2204.09715/) (2022)
- [Training Mixed-Domain Translation Models via Federated Learning](https://mlsanity.com/paper/2205.01557/) (2022)
- [Federated Split BERT for Heterogeneous Text Classification](https://mlsanity.com/paper/2205.13299/) (2022)
- [Federated Multilingual Models for Medical Transcript Analysis](https://mlsanity.com/paper/2211.09722/) (2022)
- [KNOT: Knowledge Distillation using Optimal Transport for Solving NLP Tasks](https://mlsanity.com/paper/2110.02432/) (2021)
- [FDAPT: Federated Domain-adaptive Pre-training for Language Models](https://mlsanity.com/paper/2307.06933/) (2023)
- [SoK: Privacy-aware LLM in Healthcare: Threat Model, Privacy Techniques, Challenges and Recommendations](https://mlsanity.com/paper/2601.10004/) (2026)
- [BERT WEAVER: Using WEight AVERaging to enable lifelong learning for transformer-based models in biomedical semantic search engines](https://mlsanity.com/paper/2202.10101/) (2022)
- [A Federated Approach for Hate Speech Detection](https://mlsanity.com/paper/2302.09243/) (2023)
- [An In-Depth Evaluation of Federated Learning on Biomedical Natural Language Processing](https://mlsanity.com/paper/2307.11254/) (2023)
- [FedGlu: A personalized federated learning-based glucose forecasting algorithm for improved performance in glycemic excursion regions](https://mlsanity.com/paper/2408.13926/) (2024)
- [Contrast with Major Classifier Vectors for Federated Medical Relation Extraction with Heterogeneous Label Distribution](https://mlsanity.com/paper/2301.05376/) (2023)
- [FISGuard: Defending Against Membership Inference via Fixed Input Subspaces](https://mlsanity.com/paper/2608.27836/) (2026)

## Abstract (arXiv)

Large scale contextual representation models, such as BERT, have significantly advanced natural language processing (NLP) in recently years. However, in certain area like healthcare, accessing diverse large scale text data from multiple institutions is extremely challenging due to privacy and regulatory reasons. In this article, we show that it is possible to both pretrain and fine tune BERT models in a federated manner using clinical texts from different silos without moving the data.