# Learning Private Neural Language Modeling with Attentive Aggregation

arXiv: https://arxiv.org/abs/1812.07108 · HTML: https://mlsanity.com/paper/1812.07108/

## Key facts

- Cited by 49 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Dec 2018.

## Built on by (most cited first, 30 of 49)

- [Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunities](https://mlsanity.com/paper/2408.07666/) (2024)
- [FedNLP: Benchmarking Federated Learning Methods for Natural Language Processing Tasks](https://mlsanity.com/paper/2104.08815/) (2021)
- [When Federated Learning Meets Pre-trained Language Models' Parameter-Efficient Tuning Methods](https://mlsanity.com/paper/2212.10025/) (2022)
- [Federated Continual Learning via Knowledge Fusion: A Survey](https://mlsanity.com/paper/2312.16475/) (2023)
- [Federated Learning Meets Natural Language Processing: A Survey](https://mlsanity.com/paper/2107.12603/) (2021)
- [Emerging Trends in Federated Learning: From Model Fusion to Federated X Learning](https://mlsanity.com/paper/2102.12920/) (2021)
- [Pretrained Models for Multilingual Federated Learning](https://mlsanity.com/paper/2206.02291/) (2022)
- [Personalized Federated Learning With Graph](https://mlsanity.com/paper/2203.00829/) (2022)
- [Prototype Guided Federated Learning of Visual Feature Representations](https://mlsanity.com/paper/2105.08982/) (2021)
- [Applications of Federated Learning in Smart Cities: Recent Advances, Taxonomy, and Open Challenges](https://mlsanity.com/paper/2102.01375/) (2021)
- [Fed-FSNet: Mitigating Non-I.I.D. Federated Learning via Fuzzy Synthesizing Network](https://mlsanity.com/paper/2208.12044/) (2022)
- [Mitigating Biases in Student Performance Prediction via Attention-Based Personalized Federated Learning](https://mlsanity.com/paper/2208.01182/) (2022)
- [Exploring Deep Reinforcement Learning-Assisted Federated Learning for Online Resource Allocation in Privacy-Persevering EdgeIoT](https://mlsanity.com/paper/2202.07391/) (2022)
- [Training Mixed-Domain Translation Models via Federated Learning](https://mlsanity.com/paper/2205.01557/) (2022)
- [Federated Learning and Meta Learning: Approaches, Applications, and Directions](https://mlsanity.com/paper/2210.13111/) (2022)
- [Secure and Efficient Federated Learning Through Layering and Sharding Blockchain](https://mlsanity.com/paper/2104.13130/) (2021)
- [Dynamic Attention-based Communication-Efficient Federated Learning](https://mlsanity.com/paper/2108.05765/) (2021)
- [Federated Recommendation with Additive Personalization](https://mlsanity.com/paper/2301.09109/) (2023)
- [A Survey on Vulnerability of Federated Learning: A Learning Algorithm Perspective](https://mlsanity.com/paper/2311.16065/) (2023)
- [Speaker-Conditioned Hierarchical Modeling for Automated Speech Scoring](https://mlsanity.com/paper/2109.00928/) (2021)
- [Unimodal Training-Multimodal Prediction: Cross-modal Federated Learning with Hierarchical Aggregation](https://mlsanity.com/paper/2303.15486/) (2023)
- [Promoting Data and Model Privacy in Federated Learning through Quantized LoRA](https://mlsanity.com/paper/2406.10976/) (2024)
- [An Adaptive Federated Relevance Framework for Spatial Temporal Graph Learning](https://mlsanity.com/paper/2206.03420/) (2022)
- [Multi-Layer Personalized Federated Learning for Mitigating Biases in Student Predictive Analytics](https://mlsanity.com/paper/2212.02985/) (2022)
- [Worldwide Federated Training of Language Models](https://mlsanity.com/paper/2405.14446/) (2024)
- [Federated learning framework for collaborative remaining useful life prognostics: an aircraft engine case study](https://mlsanity.com/paper/2506.00499/) (2025)
- [Private Language Model Adaptation for Speech Recognition](https://mlsanity.com/paper/2110.10026/) (2021)
- [FedBIAD: Communication-Efficient and Accuracy-Guaranteed Federated Learning with Bayesian Inference-Based Adaptive Dropout](https://mlsanity.com/paper/2307.07172/) (2023)
- [Federated Learning for 6G: Paradigms, Taxonomy, Recent Advances and Insights](https://mlsanity.com/paper/2312.04688/) (2023)
- [Cellular Traffic Prediction via Byzantine-robust Asynchronous Federated Learning](https://mlsanity.com/paper/2505.19263/) (2025)

## Abstract (arXiv)

Mobile keyboard suggestion is typically regarded as a word-level language modeling problem. Centralized machine learning technique requires massive user data collected to train on, which may impose privacy concerns for sensitive personal typing data of users. Federated learning (FL) provides a promising approach to learning private language modeling for intelligent personalized keyboard suggestion by training models in distributed clients rather than training in a central server. To obtain a global model for prediction, existing FL algorithms simply average the client models and ignore the importance of each client during model aggregation. Furthermore, there is no optimization for learning a well-generalized global model on the central server. To solve these problems, we propose a novel model aggregation with the attention mechanism considering the contribution of clients models to the global model, together with an optimization technique during server aggregation. Our proposed attentive aggregation method minimizes the weighted distance between the server model and client models through iterative parameters updating while attends the distance between the server model and client models. Through experiments on two popular language modeling datasets and a social media dataset, our proposed method outperforms its counterparts in terms of perplexity and communication cost in most settings of comparison.