mlsanityData to 14 Oct 2026

Learning Private Neural Language Modeling with Attentive Aggregation

arXiv 1812.07108 ↗First posted 17 Dec 2018cs.CL cs.LGcited by 49

Not on the map. Why?

Abstract (arXiv)

Mobile keyboard suggestion is typically regarded as a word-level language modeling problem. Centralized machine learning technique requires massive user data collected to train on, which may impose privacy concerns for sensitive personal typing data of users. Federated learning (FL) provides a promising approach to learning private language modeling for intelligent personalized keyboard suggestion by training models in distributed clients rather than training in a central server. To obtain a global model for prediction, existing FL algorithms simply average the client models and ignore the importance of each client during model aggregation. Furthermore, there is no optimization for learning a well-generalized global model on the central server. To solve these problems, we propose a novel model aggregation with the attention mechanism considering the contribution of clients models to the global model, together with an optimization technique during server aggregation. Our proposed attentive aggregation method minimizes the weighted distance between the server model and client models through iterative parameters updating while attends the distance between the server model and client models. Through experiments on two popular language modeling datasets and a social media dataset, our proposed method outperforms its counterparts in terms of perplexity and communication cost in most settings of comparison.

Key facts. Cited by 49 papers in the map (data through 14 Oct 2026). First posted on arXiv 17 Dec 2018.

Builds on

No reference list yet.

Built on by

Cited by year

Citing papers per year: 2021: 12; 2022: 12; 2023: 9; 2024: 9; 2025: 7.

49 papers.

Sort:

  1. 2024Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunitiescited by 254
  2. 2021FedNLP: Benchmarking Federated Learning Methods for Natural Language Processing Taskscited by 85
  3. 2022When Federated Learning Meets Pre-trained Language Models' Parameter-Efficient Tuning Methodscited by 75
  4. 2023Federated Continual Learning via Knowledge Fusion: A Surveycited by 53
  5. 2021Federated Learning Meets Natural Language Processing: A Surveycited by 43
  6. 2021Emerging Trends in Federated Learning: From Model Fusion to Federated X Learningcited by 35
  7. 2022Pretrained Models for Multilingual Federated Learningcited by 30
  8. 2022Personalized Federated Learning With Graphcited by 29
  9. 2021Prototype Guided Federated Learning of Visual Feature Representationscited by 22
  10. 2021Applications of Federated Learning in Smart Cities: Recent Advances, Taxonomy, and Open Challengescited by 17
  11. 2022Fed-FSNet: Mitigating Non-I.I.D. Federated Learning via Fuzzy Synthesizing Networkcited by 14
  12. 2022Mitigating Biases in Student Performance Prediction via Attention-Based Personalized Federated Learningcited by 13
37 more
  1. 2022Exploring Deep Reinforcement Learning-Assisted Federated Learning for Online Resource Allocation in Privacy-Persevering EdgeIoTcited by 12
  2. 2022Training Mixed-Domain Translation Models via Federated Learningcited by 12
  3. 2022Federated Learning and Meta Learning: Approaches, Applications, and Directionscited by 12
  4. 2021Secure and Efficient Federated Learning Through Layering and Sharding Blockchaincited by 10
  5. 2021Dynamic Attention-based Communication-Efficient Federated Learningcited by 9
  6. 2023Federated Recommendation with Additive Personalizationcited by 9
  7. 2023A Survey on Vulnerability of Federated Learning: A Learning Algorithm Perspectivecited by 9
  8. 2021Speaker-Conditioned Hierarchical Modeling for Automated Speech Scoringcited by 8
  9. 2023Unimodal Training-Multimodal Prediction: Cross-modal Federated Learning with Hierarchical Aggregationcited by 8
  10. 2024Promoting Data and Model Privacy in Federated Learning through Quantized LoRAcited by 8
  11. 2022An Adaptive Federated Relevance Framework for Spatial Temporal Graph Learningcited by 7
  12. 2022Multi-Layer Personalized Federated Learning for Mitigating Biases in Student Predictive Analyticscited by 5
  13. 2024Worldwide Federated Training of Language Modelscited by 5
  14. 2025Federated learning framework for collaborative remaining useful life prognostics: an aircraft engine case studycited by 5
  15. 2021Private Language Model Adaptation for Speech Recognitioncited by 4
  16. 2023FedBIAD: Communication-Efficient and Accuracy-Guaranteed Federated Learning with Bayesian Inference-Based Adaptive Dropoutcited by 4
  17. 2023Federated Learning for 6G: Paradigms, Taxonomy, Recent Advances and Insightscited by 4
  18. 2025Cellular Traffic Prediction via Byzantine-robust Asynchronous Federated Learningcited by 4
  19. 2021Attentive Federated Learning for Concept Drift in Distributed 5G Edge Networkscited by 3
  20. 2023FedGeo: Privacy-Preserving User Next Location Prediction with Federated Learningcited by 3
  21. 2024Unlocking the Potential of Model Calibration in Federated Learningcited by 3
  22. 2025Towards Heterogeneity-Aware and Energy-Efficient Topology Optimization for Decentralized Federated Learning in Edge Environmentcited by 3
  23. 2021Gradual Federated Learning with Simulated Annealingcited by 2
  24. 2022Online Meta-Learning for Model Update Aggregation in Federated Learning for Click-Through Rate Predictioncited by 2
  25. 2024VeCAF: Vision-language Collaborative Active Finetuning with Training Objective Awarenesscited by 2
  26. 2025FedCD: A Fairness-aware Federated Cognitive Diagnosis Frameworkcited by 2
  27. 2021SCEI: A Smart-Contract Driven Edge Intelligence Framework for IoT Systemscited by 1
  28. 2022Group Personalized Federated Learningcited by 1
  29. 2023Federated Learning with Classifier Shift for Class Imbalancecited by 1
  30. 2024Rethinking the Starting Point: Collaborative Pre-Training for Federated Downstream Taskscited by 1
  31. 2024FedASTA: Federated adaptive spatial-temporal attention for traffic flow predictioncited by 1
  32. 2025FedSEA-LLaMA: A Secure, Efficient and Adaptive Federated Splitting Framework for Large Language Modelscited by 1
  33. 2025Distribution-Controlled Client Selection to Improve Federated Learning Strategiescited by 1
  34. 2023The Landscape of Modern Machine Learning: A Review of Machine, Distributed and Federated Learningcited by 0
  35. 2024Transformers with Attentive Federated Aggregation for Time Series Stock Forecastingcited by 0
  36. 2024DUA-D2C: Dynamic Uncertainty Aware Method for Overfitting Remediation in Deep Learningcited by 0
  37. 2025Far From Sight, Far From Mind: Inverse Distance Weighting for Graph Federated Recommendationcited by 0