# Federated Learning of N-gram Language Models

arXiv: https://arxiv.org/abs/1910.03432 · HTML: https://mlsanity.com/paper/1910.03432/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Oct 2019.

## Built on by (most cited first, 30 of 33)

- [FedNLP: Benchmarking Federated Learning Methods for Natural Language Processing Tasks](https://mlsanity.com/paper/2104.08815/) (2021)
- [Federated Learning Meets Natural Language Processing: A Survey](https://mlsanity.com/paper/2107.12603/) (2021)
- [When BERT Meets Quantum Temporal Convolution Learning for Text Classification in Heterogeneous Computing](https://mlsanity.com/paper/2203.03550/) (2022)
- [Backdoor Attacks and Defenses in Federated Learning: Survey, Challenges and Future Research Directions](https://mlsanity.com/paper/2303.02213/) (2023)
- [Pretrained Models for Multilingual Federated Learning](https://mlsanity.com/paper/2206.02291/) (2022)
- [Scaling Language Model Size in Cross-Device Federated Learning](https://mlsanity.com/paper/2204.09715/) (2022)
- [The Internet of Federated Things (IoFT): A Vision for the Future and In-depth Survey of Data-driven Approaches for Federated Learning](https://mlsanity.com/paper/2111.05326/) (2021)
- [How to keep text private? A systematic review of deep learning methods for privacy-preserving natural language processing](https://mlsanity.com/paper/2205.10095/) (2022)
- [FLASHE: Additively Symmetric Homomorphic Encryption for Cross-Silo Federated Learning](https://mlsanity.com/paper/2109.00675/) (2021)
- [Auto-weighted Robust Federated Learning with Corrupted Data Sources](https://mlsanity.com/paper/2101.05880/) (2021)
- [From Deterioration to Acceleration: A Calibration Approach to Rehabilitating Step Asynchronism in Federated Optimization](https://mlsanity.com/paper/2112.09355/) (2021)
- [C2A: Client-Customized Adaptation for Parameter-Efficient Federated Learning](https://mlsanity.com/paper/2411.00311/) (2024)
- [Federated Natural Language Generation for Personalized Dialogue System](https://mlsanity.com/paper/2110.06419/) (2021)
- [Federated Nearest Neighbor Machine Translation](https://mlsanity.com/paper/2302.12211/) (2023)
- [Aggregate or Not? Exploring Where to Privatize in DNN Based Federated Learning Under Different Non-IID Scenes](https://mlsanity.com/paper/2107.11954/) (2021)
- [Bristle: Decentralized Federated Learning in Byzantine, Non-i.i.d. Environments](https://mlsanity.com/paper/2110.11006/) (2021)
- [Algorithms for bounding contribution for histogram estimation under user-level privacy](https://mlsanity.com/paper/2206.03008/) (2022)
- [A Federated Learning Approach to Privacy Preserving Offensive Language Identification](https://mlsanity.com/paper/2404.11470/) (2024)
- [Privacy in Open Search: A Review of Challenges and Solutions](https://mlsanity.com/paper/2110.10720/) (2021)
- [Capitalization Normalization for Language Modeling with an Accurate and Efficient Hierarchical RNN Model](https://mlsanity.com/paper/2202.08171/) (2022)
- [N-gram Is Back: Residual Learning of Neural Text Generation with n-gram Language Model](https://mlsanity.com/paper/2210.14431/) (2022)
- [Personalized Language Model Learning on Text Data Without User Identifiers](https://mlsanity.com/paper/2501.06062/) (2025)
- [Now It Sounds Like You: Learning Personalized Vocabulary On Device](https://mlsanity.com/paper/2305.03584/) (2023)
- [Efficient Language Model Architectures for Differentially Private Federated Learning](https://mlsanity.com/paper/2403.08100/) (2024)
- [Preliminary Steps Towards Federated Sentiment Classification](https://mlsanity.com/paper/2107.11956/) (2021)
- [Position-Invariant Truecasing with a Word-and-Character Hierarchical Recurrent Neural Network](https://mlsanity.com/paper/2108.11943/) (2021)
- [Fingerprint Attack: Client De-Anonymization in Federated Learning](https://mlsanity.com/paper/2310.05960/) (2023)
- [LLMs as On-demand Customizable Service](https://mlsanity.com/paper/2401.16577/) (2024)
- [Recent Advances in Federated Learning Driven Large Language Models: A Survey on Architecture, Performance, and Security](https://mlsanity.com/paper/2406.09831/) (2024)
- [Neural Search Space in Gboard Decoder](https://mlsanity.com/paper/2410.15575/) (2024)

## Abstract (arXiv)

We propose algorithms to train production-quality n-gram language models using federated learning. Federated learning is a distributed computation platform that can be used to train global models for portable devices such as smart phones. Federated learning is especially relevant for applications handling privacy-sensitive data, such as virtual keyboards, because training is performed without the users' data ever leaving their devices. While the principles of federated learning are fairly generic, its methodology assumes that the underlying models are neural networks. However, virtual keyboards are typically powered by n-gram language models for latency reasons. We propose to train a recurrent neural network language model using the decentralized FederatedAveraging algorithm and to approximate this federated model server-side with an n-gram model that can be deployed to devices for fast inference. Our technical contributions include ways of handling large vocabularies, algorithms to correct capitalization errors in user data, and efficient finite state transducer algorithms to convert word language models to word-piece language models and vice versa. The n-gram language models trained with federated learning are compared to n-grams trained with traditional server-based algorithms using A/B tests on tens of millions of users of virtual keyboard. Results are presented for two languages, American English and Brazilian Portuguese. This work demonstrates that high-quality n-gram language models can be trained directly on client mobile devices without sensitive training data ever leaving the devices.