mlsanityData to 7 Oct 2026

Training Large-Vocabulary Neural Language Models by Private Federated Learning for Resource-Constrained Devices

arXiv 2207.08988 ↗First posted 18 Jul 2022cs.LG cs.CL cs.CRcited by 16

Not on the map. Why?

Abstract (arXiv)

Federated Learning (FL) is a technique to train models using data distributed across devices. Differential Privacy (DP) provides a formal privacy guarantee for sensitive data. Our goal is to train a large neural network language model (NNLM) on compute-constrained devices while preserving privacy using FL and DP. However, the DP-noise introduced to the model increases as the model size grows, which often prevents convergence. We propose Partial Embedding Updates (PEU), a novel technique to decrease noise by decreasing payload size. Furthermore, we adopt Low Rank Adaptation (LoRA) and Noise Contrastive Estimation (NCE) to reduce the memory demands of large models on compute-constrained devices. This combination of techniques makes it possible to train large-vocabulary language models while preserving accuracy and privacy.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Builds on 26 of its 40 references that are in the map. First posted on arXiv 18 Jul 2022.

Builds on

26 of 40 references.

References · 26

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  5. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  6. 2016Deep Learning with Differential Privacycited by 2,682
  7. 2020Extracting Training Data from Large Language Modelscited by 2,144
  8. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  9. 2020Adaptive Federated Optimizationcited by 1,036
  10. 2019Deep Leakage from Gradientscited by 1,004
  11. 2018The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networkscited by 847
  12. 2019Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classificationcited by 836
14 more
  1. 2017Renyi Differential Privacycited by 643
  2. 2018Federated Learning for Mobile Keyboard Predictioncited by 565
  3. 2018Exploiting Unintended Feature Leakage in Collaborative Learningcited by 509
  4. 2014New insights and perspectives on the natural gradient methodcited by 435
  5. 2021Large Language Models Can Be Strong Differentially Private Learnerscited by 405
  6. 2021Differentially Private Fine-tuning of Language Modelscited by 392
  7. 2016Using the Output Embedding to Improve Language Modelscited by 266
  8. 2019R'enyi Differential Privacy of the Sampled Gaussian Mechanismcited by 199
  9. 2021Large-Scale Differentially Private BERTcited by 117
  10. 2019Federated Learning for Emoji Prediction in a Mobile Keyboardcited by 115
  11. 2014Differentially Private Empirical Risk Minimization: Efficient Algorithms and Tight Error Boundscited by 104
  12. 2020Training Production Language Models without Memorizing User Datacited by 68
  13. 2019SNDCNN: Self-normalizing deep CNNs with scaled exponential linear units for speech recognitioncited by 13
  14. 2021A Discriminative Entity-Aware Language Model for Virtual Assistantscited by 7

Built on by

Cited by year

Citing papers per year: 2022: 2; 2023: 5; 2024: 8; 2025: 1.

16 papers.

Sort:

  1. 2024Federated Fine-tuning of Large Language Models under Heterogeneous Tasks and Client Resourcescited by 93
  2. 2022When Federated Learning Meets Pre-trained Language Models' Parameter-Efficient Tuning Methodscited by 74
  3. 2023Identifying and Mitigating Privacy Risks Stemming from Language Models: A Surveycited by 30
  4. 2024The Future of Large Language Model Pre-training is Federatedcited by 30
  5. 2024PrE-Text: Training Language Models on Private Federated Data in the Age of LLMscited by 29
  6. 2023Integration of Large Language Models and Federated Learningcited by 28
  7. 2023FedBPT: Efficient Federated Black-box Prompt Tuning for Large Language Modelscited by 25
  8. 2022CANIFE: Crafting Canaries for Empirical Privacy Measurement in Federated Learningcited by 24
  9. 2024Deconstructing The Ethics of Large Language Models from Long-standing Issues to New-emerging Dilemmas: A Surveycited by 23
  10. 2024DP-DyLoRA: Fine-Tuning Transformer-Based Models On-Device under Differentially Private Federated Learning using Dynamic Low-Rank Adaptationcited by 9
  11. 2024pfl-research: simulation framework for accelerating research in Private Federated Learningcited by 8
  12. 2023Modeling Spoken Information Queries for Virtual Assistants: Open Problems, Challenges and Opportunitiescited by 4
4 more
  1. 2024Momentum Approximation in Asynchronous Private Federated Learningcited by 3
  2. 2023Jointly Reparametrized Multi-Layer Adaptation for Efficient and Private Tuningcited by 2
  3. 2024Enhancing CTC-based speech recognition with diverse modeling unitscited by 1
  4. 2025Private Federated Multiclass Post-hoc Calibrationcited by 0