# Federated Learning Meets Natural Language Processing: A Survey

arXiv: https://arxiv.org/abs/2107.12603 · HTML: https://mlsanity.com/paper/2107.12603/

## Key facts

- Cited by 43 papers in the map (data through 14 Oct 2026).
- Builds on 41 of its 72 references that are in the map.
- First posted on arXiv 27 Jul 2021.

## Builds on (most cited first, 30 of 41)

- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [Advances and Open Problems in Federated Learning](https://mlsanity.com/paper/1912.04977/) (2019)
- [Federated Learning with Non-IID Data](https://mlsanity.com/paper/1806.00582/) (2018)
- [Ensemble Distillation for Robust Model Fusion in Federated Learning](https://mlsanity.com/paper/2006.07242/) (2020)
- [Federated Learning for Mobile Keyboard Prediction](https://mlsanity.com/paper/1811.03604/) (2018)
- [Federated Learning with Differential Privacy: Algorithms and Performance Analysis](https://mlsanity.com/paper/1911.00222/) (2019)
- [LSTM: A Search Space Odyssey](https://mlsanity.com/paper/1503.04069/) (2015)
- [Personalized Federated Learning: A Meta-Learning Approach](https://mlsanity.com/paper/2002.07948/) (2020)
- [The Non-IID Data Quagmire of Decentralized Machine Learning](https://mlsanity.com/paper/1910.00189/) (2019)
- [Dataset Distillation](https://mlsanity.com/paper/1811.10959/) (2018)
- [Applied Federated Learning: Improving Google Keyboard Query Suggestions](https://mlsanity.com/paper/1812.02903/) (2018)
- [SecureBoost: A Lossless Federated Learning Framework](https://mlsanity.com/paper/1901.08755/) (2019)
- [One-Shot Federated Learning](https://mlsanity.com/paper/1902.11175/) (2019)
- [Federated Learning for Emoji Prediction in a Mobile Keyboard](https://mlsanity.com/paper/1906.04329/) (2019)
- [Hello Edge: Keyword Spotting on Microcontrollers](https://mlsanity.com/paper/1711.07128/) (2017)
- [Distilled One-Shot Federated Learning](https://mlsanity.com/paper/2009.07999/) (2020)
- [Soft-Label Dataset Distillation and Text Dataset Distillation](https://mlsanity.com/paper/1910.02551/) (2019)
- [Federated Learning for Keyword Spotting](https://mlsanity.com/paper/1810.05512/) (2018)
- [Federated Collaborative Filtering for Privacy-Preserving Personalized Recommendation System](https://mlsanity.com/paper/1901.09888/) (2019)
- [Patient Clustering Improves Efficiency of Federated Machine Learning to predict mortality and hospital stay time using distributed Electronic Medical Records](https://mlsanity.com/paper/1903.09296/) (2019)
- [FedHome: Cloud-Edge based Personalized Federated Learning for In-Home Health Monitoring](https://mlsanity.com/paper/2012.07450/) (2020)
- [Decentralizing Feature Extraction with Quantum Convolutional Neural Network for Automatic Speech Recognition](https://mlsanity.com/paper/2010.13309/) (2020)
- [Federated Learning Of Out-Of-Vocabulary Words](https://mlsanity.com/paper/1903.10635/) (2019)
- [Learning Private Neural Language Modeling with Attentive Aggregation](https://mlsanity.com/paper/1812.07108/) (2018)
- [Training Speech Recognition Models with Federated Learning: A Quality/Cost Framework](https://mlsanity.com/paper/2010.15965/) (2020)
- [Understanding Unintended Memorization in Federated Learning](https://mlsanity.com/paper/2006.07490/) (2020)
- [Scaling Federated Learning for Fine-tuning of Large Language Models](https://mlsanity.com/paper/2102.00875/) (2021)
- [Federated Learning of N-gram Language Models](https://mlsanity.com/paper/1910.03432/) (2019)

## Built on by (most cited first, 30 of 43)

- [Towards Building the Federated GPT: Federated Instruction Tuning](https://mlsanity.com/paper/2305.05644/) (2023)
- [Advances and Open Challenges in Federated Foundation Models](https://mlsanity.com/paper/2404.15381/) (2024)
- [Pretrained Models for Multilingual Federated Learning](https://mlsanity.com/paper/2206.02291/) (2022)
- [Towards Quantum Federated Learning](https://mlsanity.com/paper/2306.09912/) (2023)
- [Privacy enabled Financial Text Classification using Differential Privacy and Federated Learning](https://mlsanity.com/paper/2110.01643/) (2021)
- [Federated Optimization Algorithms with Random Reshuffling and Gradient Compression](https://mlsanity.com/paper/2206.07021/) (2022)
- [Faster federated optimization under second-order similarity](https://mlsanity.com/paper/2209.02257/) (2022)
- [Beyond Exponentially Fast Mixing in Average-Reward Reinforcement Learning via Multi-Level Monte Carlo Actor-Critic](https://mlsanity.com/paper/2301.12083/) (2023)
- [Mutual Enhancement of Large and Small Language Models with Cross-Silo Knowledge Transfer](https://mlsanity.com/paper/2312.05842/) (2023)
- [Federated Learning and Meta Learning: Approaches, Applications, and Directions](https://mlsanity.com/paper/2210.13111/) (2022)
- [Open Challenges and Opportunities in Federated Foundation Models Towards Biomedical Healthcare](https://mlsanity.com/paper/2405.06784/) (2024)
- [Federated Multilingual Models for Medical Transcript Analysis](https://mlsanity.com/paper/2211.09722/) (2022)
- [Stochastic Distributed Optimization under Average Second-order Similarity: Algorithms and Analysis](https://mlsanity.com/paper/2304.07504/) (2023)
- [A Review of Federated Learning in Energy Systems](https://mlsanity.com/paper/2208.10941/) (2022)
- [A Survey on Vulnerability of Federated Learning: A Learning Algorithm Perspective](https://mlsanity.com/paper/2311.16065/) (2023)
- [Blockchain for Large Language Model Security and Safety: A Holistic Survey](https://mlsanity.com/paper/2407.20181/) (2024)
- [Hazards from Increasingly Accessible Fine-Tuning of Downloadable Foundation Models](https://mlsanity.com/paper/2312.14751/) (2023)
- [Adapter-based Selective Knowledge Distillation for Federated Multi-domain Meeting Summarization](https://mlsanity.com/paper/2308.03275/) (2023)
- [Multiplayer Federated Learning: Reaching Equilibrium with Less Communication](https://mlsanity.com/paper/2501.08263/) (2025)
- [Open-Vocabulary Federated Learning with Multimodal Prototyping](https://mlsanity.com/paper/2404.01232/) (2024)
- [FewFedWeight: Few-shot Federated Learning Framework across Multiple NLP Tasks](https://mlsanity.com/paper/2212.08354/) (2022)
- [Edge-assisted U-Shaped Split Federated Learning with Privacy-preserving for Internet of Things](https://mlsanity.com/paper/2311.04944/) (2023)
- [MedLeak: Multimodal Medical Data Leakage in Secure Federated Learning with Crafted Models](https://mlsanity.com/paper/2407.09972/) (2024)
- [Now It Sounds Like You: Learning Personalized Vocabulary On Device](https://mlsanity.com/paper/2305.03584/) (2023)
- [Only Send What You Need: Learning to Communicate Efficiently in Federated Multilingual Machine Translation](https://mlsanity.com/paper/2401.07456/) (2024)
- [Collaborative Visual Place Recognition through Federated Learning](https://mlsanity.com/paper/2404.13324/) (2024)
- [Federated Learning-Enabled Hybrid Language Models for Communication-Efficient Token Transmission](https://mlsanity.com/paper/2507.00082/) (2025)
- [Efficient Federated Learning with Timely Update Dissemination](https://mlsanity.com/paper/2507.06031/) (2025)
- [Federated Named Entity Recognition](https://mlsanity.com/paper/2203.15101/) (2022)
- [FLVoogd: Robust And Privacy Preserving Federated Learning](https://mlsanity.com/paper/2207.00428/) (2022)

## Abstract (arXiv)

Federated Learning aims to learn machine learning models from multiple decentralized edge devices (e.g. mobiles) or servers without sacrificing local data privacy. Recent Natural Language Processing techniques rely on deep learning and large pre-trained language models. However, both big deep neural and language models are trained with huge amounts of data which often lies on the server side. Since text data is widely originated from end users, in this work, we look into recent NLP models and techniques which use federated learning as the learning framework. Our survey discusses major challenges in federated natural language processing, including the algorithm challenges, system challenges as well as the privacy issues. We also provide a critical review of the existing Federated NLP evaluation methods and tools. Finally, we highlight the current research gaps and future directions.