Federated pretraining and fine tuning of BERT using clinical notes from multiple silos
Not on the map. Why?
Abstract (arXiv)
Large scale contextual representation models, such as BERT, have significantly advanced natural language processing (NLP) in recently years. However, in certain area like healthcare, accessing diverse large scale text data from multiple institutions is extremely challenging due to privacy and regulatory reasons. In this article, we show that it is possible to both pretrain and fine tune BERT models in a federated manner using clinical texts from different silos without moving the data.
Key facts. Cited by 18 papers in the map (data through 7 Oct 2026). First posted on arXiv 20 Feb 2020.
Builds on
No reference list yet.
Built on by
Citing papers per year: 2021: 4; 2022: 7; 2023: 4; 2024: 1; 2025: 0; 2026: 2.
18 papers.
Sort:
By area:
- 2021FedNLP: Benchmarking Federated Learning Methods for Natural Language Processing Taskscited by 84
- 2022Recovering Private Text in Federated Learning of Language Modelscited by 67
- 2021Federated Learning Meets Natural Language Processing: A Surveycited by 42
- 2021Scaling Federated Learning for Fine-tuning of Large Language Modelscited by 36
- 2022Pretrained Models for Multilingual Federated Learningcited by 30
- 2022Scaling Language Model Size in Cross-Device Federated Learningcited by 25
- 2022Training Mixed-Domain Translation Models via Federated Learningcited by 12
- 2022Federated Split BERT for Heterogeneous Text Classificationcited by 11
- 2022Federated Multilingual Models for Medical Transcript Analysiscited by 11
- 2021KNOT: Knowledge Distillation using Optimal Transport for Solving NLP Taskscited by 8
- 2023FDAPT: Federated Domain-adaptive Pre-training for Language Modelscited by 5
- 2022BERT WEAVER: Using WEight AVERaging to enable lifelong learning for transformer-based models in biomedical semantic search enginescited by 2
6 more
- 2023A Federated Approach for Hate Speech Detectioncited by 1
- 2023An In-Depth Evaluation of Federated Learning on Biomedical Natural Language Processingcited by 1
- 2024FedGlu: A personalized federated learning-based glucose forecasting algorithm for improved performance in glycemic excursion regionscited by 1
- 2026SoK: Privacy-aware LLM in Healthcare: Threat Model, Privacy Techniques, Challenges and Recommendationscited by 1
- 2023Contrast with Major Classifier Vectors for Federated Medical Relation Extraction with Heterogeneous Label Distributioncited by 0
- 2026FISGuard: Defending Against Membership Inference via Fixed Input Subspacescited by 0
- 2026FISGuard: Defending Against Membership Inference via Fixed Input Subspaces28 Aug 2026
- 2026SoK: Privacy-aware LLM in Healthcare: Threat Model, Privacy Techniques, Challenges and Recommendations15 Jan 2026
- 2024FedGlu: A personalized federated learning-based glucose forecasting algorithm for improved performance in glycemic excursion regions25 Aug 2024
- 2023An In-Depth Evaluation of Federated Learning on Biomedical Natural Language Processing20 Jul 2023
- 2023FDAPT: Federated Domain-adaptive Pre-training for Language Models12 Jul 2023
- 2023A Federated Approach for Hate Speech Detection18 Feb 2023
- 2023Contrast with Major Classifier Vectors for Federated Medical Relation Extraction with Heterogeneous Label Distribution13 Jan 2023
- 2022Federated Multilingual Models for Medical Transcript Analysis4 Nov 2022
- 2022Pretrained Models for Multilingual Federated Learning6 Jun 2022
- 2022Federated Split BERT for Heterogeneous Text Classification26 May 2022
- 2022Recovering Private Text in Federated Learning of Language Models17 May 2022
- 2022Training Mixed-Domain Translation Models via Federated Learning3 May 2022
6 more
- 2022Scaling Language Model Size in Cross-Device Federated Learning31 Mar 2022
- 2022BERT WEAVER: Using WEight AVERaging to enable lifelong learning for transformer-based models in biomedical semantic search engines21 Feb 2022
- 2021KNOT: Knowledge Distillation using Optimal Transport for Solving NLP Tasks6 Oct 2021
- 2021Federated Learning Meets Natural Language Processing: A Survey27 Jul 2021
- 2021FedNLP: Benchmarking Federated Learning Methods for Natural Language Processing Tasks18 Apr 2021
- 2021Scaling Federated Learning for Fine-tuning of Large Language Models1 Feb 2021
- 2021FedNLP: Benchmarking Federated Learning Methods for Natural Language Processing Taskscited by 84
- 2022Recovering Private Text in Federated Learning of Language Modelscited by 67
- 2021Federated Learning Meets Natural Language Processing: A Surveycited by 42
- 2021Scaling Federated Learning for Fine-tuning of Large Language Modelscited by 36
- 2022Pretrained Models for Multilingual Federated Learningcited by 30
- 2022Scaling Language Model Size in Cross-Device Federated Learningcited by 25
- 2022Training Mixed-Domain Translation Models via Federated Learningcited by 12
- 2022Federated Split BERT for Heterogeneous Text Classificationcited by 11
- 2022Federated Multilingual Models for Medical Transcript Analysiscited by 11
- 2021KNOT: Knowledge Distillation using Optimal Transport for Solving NLP Taskscited by 8
- 2023FDAPT: Federated Domain-adaptive Pre-training for Language Modelscited by 5
- 2022BERT WEAVER: Using WEight AVERaging to enable lifelong learning for transformer-based models in biomedical semantic search enginescited by 2
12 most cited of 17 All 17 (JSON)
- 2026SoK: Privacy-aware LLM in Healthcare: Threat Model, Privacy Techniques, Challenges and Recommendations15 Jan 2026
- 2024FedGlu: A personalized federated learning-based glucose forecasting algorithm for improved performance in glycemic excursion regions25 Aug 2024
- 2023An In-Depth Evaluation of Federated Learning on Biomedical Natural Language Processing20 Jul 2023
- 2023FDAPT: Federated Domain-adaptive Pre-training for Language Models12 Jul 2023
- 2023A Federated Approach for Hate Speech Detection18 Feb 2023
- 2023Contrast with Major Classifier Vectors for Federated Medical Relation Extraction with Heterogeneous Label Distribution13 Jan 2023
- 2022Federated Multilingual Models for Medical Transcript Analysis4 Nov 2022
- 2022Pretrained Models for Multilingual Federated Learning6 Jun 2022
- 2022Federated Split BERT for Heterogeneous Text Classification26 May 2022
- 2022Recovering Private Text in Federated Learning of Language Models17 May 2022
- 2022Training Mixed-Domain Translation Models via Federated Learning3 May 2022
- 2022Scaling Language Model Size in Cross-Device Federated Learning31 Mar 2022
12 newest of 17 All 17 (JSON)