mlsanityData to 7 Oct 2026

Training Production Language Models without Memorizing User Data

arXiv 2009.10031 ↗First posted 21 Sep 2020cs.LG cs.CR stat.MLcited by 68

Not on the map. Why?

Abstract (arXiv)

This paper presents the first consumer-scale next-word prediction (NWP) model trained with Federated Learning (FL) while leveraging the Differentially Private Federated Averaging (DP-FedAvg) technique. There has been prior work on building practical FL infrastructure, including work demonstrating the feasibility of training language models on mobile devices using such infrastructure. It has also been shown (in simulations on a public corpus) that it is possible to train NWP models with user-level differential privacy using the DP-FedAvg algorithm. Nevertheless, training production-quality NWP models with DP-FedAvg in a real-world production environment on a heterogeneous fleet of mobile phones requires addressing numerous challenges. For instance, the coordinating central server has to keep track of the devices available at the start of each round and sample devices uniformly at random from them, while ensuring secrecy of the sample, etc. Unlike all prior privacy-focused FL work of which we are aware, for the first time we demonstrate the deployment of a differentially private mechanism for the training of a production neural network in FL, as well as the instrumentation of the production training infrastructure to perform an end-to-end empirical measurement of unintended memorization.

Key facts. Cited by 68 papers in the map (data through 7 Oct 2026). First posted on arXiv 21 Sep 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 15; 2022: 21; 2023: 14; 2024: 8; 2025: 9; 2026: 1.

68 papers.

Sort:

By area:

  1. 2021Ethical and social risks of harm from Language Modelscited by 957
  2. 2022Quantifying Memorization Across Neural Language Modelscited by 755
  3. 2021Large Language Models Can Be Strong Differentially Private Learnerscited by 405
  4. 2021Differentially Private Fine-tuning of Language Modelscited by 392
  5. 2022What Does it Mean for a Language Model to Preserve Privacy?cited by 194
  6. 2021Practical and Private (Deep) Learning without Sampling or Shufflingcited by 169
  7. 2021When the Curious Abandon Honesty: Federated Learning Is Not Privatecited by 126
  8. 2022Preventing Verbatim Memorization in Language Models Gives a False Sense of Privacycited by 97
  9. 2022Truth Serum: Poisoning Machine Learning Models to Reveal Their Secretscited by 94
  10. 2021The Skellam Mechanism for Differentially Private Federated Learningcited by 66
  11. 2021Eluding Secure Aggregation in Federated Learning via Model Inconsistencycited by 66
  12. 2023Federated Learning of Gboard Language Models with Differential Privacycited by 64
38 more
  1. 2021Selective Differential Privacy for Language Modelingcited by 61
  2. 2022Improved Differential Privacy for SGD via Optimal Private Linear Operators on Adaptive Streamscited by 59
  3. 2024PAPILLON: Privacy Preservation from Internet-based and Local Language Model Ensemblescited by 54
  4. 2024Operationalizing a Threat Model for Red-Teaming Large Language Models (LLMs)cited by 44
  5. 2023Training Data Extraction From Pre-trained Language Models: A Surveycited by 43
  6. 2022On Privacy and Personalization in Cross-Silo Federated Learningcited by 36
  7. 2023User Inference Attacks on Large Language Modelscited by 36
  8. 2022Just Fine-tune Twice: Selective Differential Privacy for Large Language Modelscited by 32
  9. 2022Submix: Practical Private Prediction for Large-Scale Language Modelscited by 31
  10. 2023Can Public Large Language Models Help Private Cross-device Federated Learning?cited by 30
  11. 2023Identifying and Mitigating Privacy Risks Stemming from Language Models: A Surveycited by 30
  12. 2021Privacy Regularization: Joint Privacy-Utility Optimization in Language Modelscited by 29
  13. 2024PrE-Text: Training Language Models on Private Federated Data in the Age of LLMscited by 29
  14. 2024Undesirable Memorization in Large Language Models: A Surveycited by 29
  15. 2021Resource-Efficient Federated Learningcited by 26
  16. 2022TCT: Convexifying Federated Learning using Bootstrapped Neural Tangent Kernelscited by 26
  17. 2021The Role of Adaptive Optimizers for Honest Private Hyperparameter Selectioncited by 25
  18. 2022Scaling Language Model Size in Cross-Device Federated Learningcited by 25
  19. 2022CANIFE: Crafting Canaries for Empirical Privacy Measurement in Federated Learningcited by 24
  20. 2022Learning to Generate Image Embeddings with User-level Differential Privacycited by 23
  21. 2023Differentially Private Natural Language Models: Recent Advances and Future Directionscited by 20
  22. 2024Privacy Backdoors: Stealing Data with Corrupted Pretrained Modelscited by 20
  23. 2021Wide Network Learning with Differential Privacycited by 17
  24. 2022Training Large-Vocabulary Neural Language Models by Private Federated Learning for Resource-Constrained Devicescited by 16
  25. 2023Reconstructing Individual Data Points in Federated Learning Hardened with Differential Privacy and Secure Aggregationcited by 16
  26. 2021Efficient and Private Federated Learning with Partially Trainable Networkscited by 13
  27. 2024REVS: Unlearning Sensitive Information in Language Models via Rank Editing in the Vocabulary Spacecited by 12
  28. 2022Towards Energy-Aware Federated Learning on Battery-Powered Clientscited by 11
  29. 2023Synthetic Query Generation for Privacy-Preserving Deep Retrieval Systems using Differentially Private Language Modelscited by 10
  30. 2022Detecting Unintended Memorization in Language-Model-Fused ASRcited by 9
  31. 2025From Superficial Outputs to Superficial Learning: Risks of Large Language Models in Educationcited by 9
  32. 2022Mixed Federated Learning: Joint Decentralized and Centralized Learningcited by 8
  33. 2021Jointly Learning from Decentralized (Federated) and Centralized Data to Mitigate Distribution Shiftcited by 7
  34. 2022Extracting Targeted Training Data from ASR Models, and How to Mitigate Itcited by 7
  35. 2023Optimal Locally Private Nonparametric Classification with Public Datacited by 7
  36. 2023A Note On Interpreting Canary Exposurecited by 6
  37. 2023Forgetting Private Textual Sequences in Language Models via Leave-One-Out Ensemblecited by 6
  38. 2025Mitigating Unintended Memorization with LoRA in Federated Learning for LLMscited by 6

50 most cited of 68 All 68 (JSON)