mlsanityData to 7 Oct 2026

Learning Differentially Private Recurrent Language Models

arXiv 1710.06963 ↗First posted 18 Oct 2017cs.LGcited by 32

Not on the map. Why?

Abstract (arXiv)

We demonstrate that it is possible to train large recurrent language models with user-level differential privacy guarantees with only a negligible cost in predictive accuracy. Our work builds on recent advances in the training of deep networks on user-partitioned data and privacy accounting for stochastic gradient descent. In particular, we add user-level privacy protection to the federated averaging algorithm, which makes "large step" updates from user-level data. Our work demonstrates that given a dataset with a sufficiently large number of users (a requirement easily met by even small internet-scale datasets), achieving differential privacy comes at the cost of increased computation, rather than in decreased utility as in most prior work. We find that our private LSTM language models are quantitatively and qualitatively similar to un-noised models when trained on a large dataset.

Key facts. Cited by 32 papers in the map (data through 7 Oct 2026). First posted on arXiv 18 Oct 2017.

Builds on

No reference list yet.

Built on by

Cited by year

Citing papers per year: 2021: 8; 2022: 4; 2023: 5; 2024: 9; 2025: 6.

32 papers.

Sort:

By area:

  1. 2022What Does it Mean for a Language Model to Preserve Privacy?cited by 194
  2. 2021Large-Scale Differentially Private BERTcited by 117
  3. 2021Public Data-Assisted Mirror Descent for Private Model Trainingcited by 49
  4. 2023(Amplified) Banded Matrix Factorization: A unified approach to private trainingcited by 49
  5. 2021Survey: Leakage and Privacy at Inference Timecited by 44
  6. 2022CrowdGuard: Federated Backdoor Detection in Federated Learningcited by 26
  7. 2024Federated Unlearning: A Survey on Methods, Design Guidelines, and Evaluation Metricscited by 25
  8. 2024DP-MLM: Differentially Private Text Rewriting Using Masked Language Modelscited by 24
  9. 2023BayBFed: Bayesian Backdoor Defense for Federated Learningcited by 23
  10. 2021Applications of Federated Learning in Smart Cities: Recent Advances, Taxonomy, and Open Challengescited by 17
  11. 2021Generative Adversarial Networks: A Survey Towards Private and Secure Applicationscited by 15
  12. 2021Towards Causal Federated Learning For Enhanced Robustness and Privacycited by 14
20 more
  1. 2024A Survey on Intelligent Internet of Things: Applications, Security, Privacy, and Future Directionscited by 10
  2. 2024Federated Learning under Periodic Client Participation and Heterogeneous Data: A New Communication-Efficient Algorithm and Analysiscited by 8
  3. 2025SafeSplit: A Novel Defense Against Client-Side Backdoor Attacks in Split Learning (Full Version)cited by 7
  4. 2022Hierarchical Federated Learning with Privacycited by 4
  5. 2025RewardDS: Privacy-Preserving Fine-Tuning for Large Language Models via Reward Driven Data Synthesiscited by 3
  6. 2021Utility Fairness for the Differentially Private Federated Learningcited by 2
  7. 2023LatticeGen: A Cooperative Framework which Hides Generated Text in a Lattice for Privacy-Aware Generation on Cloudcited by 2
  8. 2023Privacy-preserving design of graph neural networks with applications to vertical federated learningcited by 2
  9. 2024Differentially Private Federated Learning: Servers Trustworthiness, Estimation, and Statistical Inferencecited by 2
  10. 2024Federated Learning for Blind Image Super-Resolutioncited by 2
  11. 2024QuanCrypt-FL: Quantized Homomorphic Encryption with Pruning for Secure Federated Learningcited by 2
  12. 2025Model Inversion Attacks on Llama 3: Extracting PII from Large Language Modelscited by 2
  13. 2021Private Cross-Silo Federated Learning for Extracting Vaccine Adverse Event Mentionscited by 1
  14. 2023Avoid Adversarial Adaption in Federated Learning by Multi-Metric Investigationscited by 1
  15. 2024Can large language models be privacy preserving and fair medical coders?cited by 1
  16. 2025Enhancing Privacy in the Early Detection of Sexual Predators Through Federated Learning and Differential Privacycited by 1
  17. 2022Protecting Data from all Parties: Combining FHE and DP in Federated Learningcited by 0
  18. 2024Upcycling Noise for Federated Unlearningcited by 0
  19. 2025Structure-Preference Enabled Graph Embedding Generation under Differential Privacycited by 0
  20. 2025TrojanDam: Detection-Free Backdoor Defense in Federated Learning through Proactive Model Robustification utilizing OOD Datacited by 0