# Pretraining Federated Text Models for Next Word Prediction

arXiv: https://arxiv.org/abs/2005.04828 · HTML: https://mlsanity.com/paper/2005.04828/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 May 2020.

## Built on by (most cited first, 22 of 22)

- [FedNLP: Benchmarking Federated Learning Methods for Natural Language Processing Tasks](https://mlsanity.com/paper/2104.08815/) (2021)
- [On the Importance and Applicability of Pre-Training for Federated Learning](https://mlsanity.com/paper/2206.11488/) (2022)
- [Where to Begin? On the Impact of Pre-Training and Initialization in Federated Learning](https://mlsanity.com/paper/2210.08090/) (2022)
- [Federated Learning Meets Natural Language Processing: A Survey](https://mlsanity.com/paper/2107.12603/) (2021)
- [Pretrained Models for Multilingual Federated Learning](https://mlsanity.com/paper/2206.02291/) (2022)
- [Scaling Language Model Size in Cross-Device Federated Learning](https://mlsanity.com/paper/2204.09715/) (2022)
- [FedAdapter: Efficient Federated Learning for Modern NLP](https://mlsanity.com/paper/2205.10162/) (2022)
- [Mutual Enhancement of Large and Small Language Models with Cross-Silo Knowledge Transfer](https://mlsanity.com/paper/2312.05842/) (2023)
- [Training Mixed-Domain Translation Models via Federated Learning](https://mlsanity.com/paper/2205.01557/) (2022)
- [On the Efficacy of Differentially Private Few-shot Image Classification](https://mlsanity.com/paper/2302.01190/) (2023)
- [Efficient Federated Prompt Tuning for Black-box Large Pre-trained Models](https://mlsanity.com/paper/2310.03123/) (2023)
- [Federated Multilingual Models for Medical Transcript Analysis](https://mlsanity.com/paper/2211.09722/) (2022)
- [Analysis of Privacy Leakage in Federated Large Language Models](https://mlsanity.com/paper/2403.04784/) (2024)
- [Federated Natural Language Generation for Personalized Dialogue System](https://mlsanity.com/paper/2110.06419/) (2021)
- [Adaptive Parameter-Efficient Federated Fine-Tuning on Heterogeneous Devices](https://mlsanity.com/paper/2412.20004/) (2024)
- [FedBIAD: Communication-Efficient and Accuracy-Guaranteed Federated Learning with Bayesian Inference-Based Adaptive Dropout](https://mlsanity.com/paper/2307.07172/) (2023)
- [Project Florida: Federated Learning Made Easy](https://mlsanity.com/paper/2307.11899/) (2023)
- [Federated Named Entity Recognition](https://mlsanity.com/paper/2203.15101/) (2022)
- [Where to Begin? On the Impact of Pre-Training and Initialization in Federated Learning](https://mlsanity.com/paper/2206.15387/) (2022)
- [SPARTAN: Sparse Hierarchical Memory for Parameter-Efficient Transformers](https://mlsanity.com/paper/2211.16634/) (2022)
- [FedSEA-LLaMA: A Secure, Efficient and Adaptive Federated Splitting Framework for Large Language Models](https://mlsanity.com/paper/2505.15683/) (2025)
- [Privacy-Preserving Quantized Federated Learning with Diverse Precision](https://mlsanity.com/paper/2507.00920/) (2025)

## Abstract (arXiv)

Federated learning is a decentralized approach for training models on distributed devices, by summarizing local changes and sending aggregate parameters from local models to the cloud rather than the data itself. In this research we employ the idea of transfer learning to federated training for next word prediction (NWP) and conduct a number of experiments demonstrating enhancements to current baselines for which federated NWP models have been successful. Specifically, we compare federated training baselines from randomly initialized models to various combinations of pretraining approaches including pretrained word embeddings and whole model pretraining followed by federated fine tuning for NWP on a dataset of Stack Overflow posts. We realize lift in performance using pretrained embeddings without exacerbating the number of required training rounds or memory footprint. We also observe notable differences using centrally pretrained networks, especially depending on the datasets used. Our research offers effective, yet inexpensive, improvements to federated NWP and paves the way for more rigorous experimentation of transfer learning techniques for federated learning.