mlsanityData to 14 Oct 2026

Can Public Large Language Models Help Private Cross-device Federated Learning?

arXiv 2305.12132 ↗First posted 20 May 2023cs.LGcited by 30

Not on the map. Why?

Abstract (arXiv)

We study (differentially) private federated learning (FL) of language models. The language models in cross-device FL are relatively small, which can be trained with meaningful formal user-level differential privacy (DP) guarantees when massive parallelism in training is enabled by the participation of a moderate size of users. Recently, public data has been used to improve privacy-utility trade-offs for both large and small language models. In this work, we provide a systematic study of using large-scale public data and LLMs to help differentially private training of on-device FL models, and further improve the privacy-utility tradeoff by techniques of distillation. Moreover, we propose a novel distribution matching algorithm with theoretical grounding to sample public data close to private data distribution, which significantly improves the sample efficiency of (pre-)training on public data. The proposed method is efficient and effective for training private models by taking advantage of public data, especially for customized on-device architectures that do not have ready-to-use pre-trained models.

Key facts. Cited by 30 papers in the map (data through 14 Oct 2026). Builds on 37 of its 60 references that are in the map. First posted on arXiv 20 May 2023.

Builds on

37 of 60 references.

References · 37

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  6. 2019Advances and Open Problems in Federated Learningcited by 3,046
  7. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
  8. 2016Deep Learning with Differential Privacycited by 2,728
  9. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,989
  10. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,463
  11. 2022LaMDA: Language Models for Dialog Applicationscited by 1,220
  12. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,167
25 more
  1. 2020Adaptive Federated Optimizationcited by 1,046
  2. 2018Federated Learning for Mobile Keyboard Predictioncited by 572
  3. 2020MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devicescited by 482
  4. 2020Dataset Cartography: Mapping and Diagnosing Datasets with Training Dynamicscited by 425
  5. 2021Large Language Models Can Be Strong Differentially Private Learnerscited by 412
  6. 2021Differentially Private Fine-tuning of Language Modelscited by 400
  7. 2021A Field Guide to Federated Optimizationcited by 315
  8. 2019Depth-Adaptive Transformercited by 236
  9. 2019Differentially Private Learning with Adaptive Clippingcited by 220
  10. 2020Compressing BERT: Studying the Effects of Weight Pruning on Transfer Learningcited by 185
  11. 2021Practical and Private (Deep) Learning without Sampling or Shufflingcited by 170
  12. 2022Where to Begin? On the Impact of Pre-Training and Initialization in Federated Learningcited by 71
  13. 2020Training Production Language Models without Memorizing User Datacited by 68
  14. 2017Automated Scalable Bayesian Inference via Hilbert Coresetscited by 55
  15. 2019Federated Learning Of Out-Of-Vocabulary Wordscited by 51
  16. 2021Public Data-Assisted Mirror Descent for Private Model Trainingcited by 49
  17. 2022Differentially Private Bias-Term Fine-tuning of Foundation Modelscited by 48
  18. 2020Differentially Private Language Models Benefit from Public Pre-trainingcited by 47
  19. 2022Private Adaptive Optimization with Side Informationcited by 38
  20. 2023Why Is Public Pretraining Necessary for Private Model Training?cited by 37
  21. 2020Training Keyword Spotting Models on Non-IID Data with Federated Learningcited by 36
  22. 2022Motley: Benchmarking Heterogeneity and Personalization in Federated Learningcited by 31
  23. 2022Learning to Generate Image Embeddings with User-level Differential Privacycited by 23
  24. 2022Federated Select: A Primitive for Communication- and Memory-Efficient Federated Learningcited by 11
  25. 2022Training a Tokenizer for Free with Private Federated Learningcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 8; 2024: 14; 2025: 6; 2026: 2.

30 papers.

Sort:

By area:

  1. 2024OpenFedLLM: Training Large Language Models on Decentralized Private Data via Federated Learningcited by 128
  2. 2024Federated Fine-tuning of Large Language Models under Heterogeneous Tasks and Client Resourcescited by 93
  3. 2023Federated Learning of Gboard Language Models with Differential Privacycited by 66
  4. 2023FATE-LLM: A Industrial Grade Federated Learning Framework for Large Language Modelscited by 63
  5. 2024SplitLoRA: A Split Parameter-Efficient Fine-Tuning Framework for Large Language Modelscited by 62
  6. 2023Privacy in Large Language Models: Attacks, Defenses and Future Directionscited by 51
  7. 2024Privacy-Preserving Instructions for Aligning Large Language Modelscited by 35
  8. 2024The Future of Large Language Model Pre-training is Federatedcited by 30
  9. 2024PrE-Text: Training Language Models on Private Federated Data in the Age of LLMscited by 29
  10. 2024Federated Large Language Models: Current Progress and Future Directionscited by 27
  11. 2023FwdLLM: Efficient FedLLM using Forward Gradientcited by 26
  12. 2023GPT-FL: Generative Pre-trained Model-Assisted Federated Learningcited by 24
18 more
  1. 2023Grounding Foundation Models through Federated Transfer Learning: A General Frameworkcited by 24
  2. 2024Prompt Public Large Language Models to Synthesize Data for Private On-device Applicationscited by 17
  3. 2024Federated Distillation: A Surveycited by 15
  4. 2023Profit: Benchmarking Personalization and Robustness Trade-off in Federated Prompt Tuningcited by 9
  5. 2024Analysis of Privacy Leakage in Federated Large Language Modelscited by 9
  6. 2024A Hassle-free Algorithm for Private Learning in Practice: Don't Use Tree Aggregation, Use BLTscited by 9
  7. 2025Privacy-Preserving Federated Embedding Learning for Localized Retrieval-Augmented Generationcited by 8
  8. 2025AugFL: Augmenting Federated Learning with Pretrained Modelscited by 6
  9. 2025Personalized Federated Fine-tuning for Heterogeneous Data: An Automatic Rank Learning Approach via Two-Level LoRAcited by 6
  10. 2025Towards Harnessing the Collaborative Power of Large and Small Models for Domain Taskscited by 4
  11. 2025Research in Collaborative Learning Does Not Serve Cross-Silo Federated Learning in Practicecited by 3
  12. 2024Federated Domain-Specific Knowledge Transfer on Large Language Models Using Synthetic Datacited by 2
  13. 2023Personalized Federated Learning via ADMM with Moreau Envelopecited by 0
  14. 2024Federated Learning with New Knowledge: Fundamentals, Advances, and Futurescited by 0
  15. 2024On ADMM in Heterogeneous Federated Learning: Personalization, Robustness, and Fairnesscited by 0
  16. 2025A Scenario-Oriented Survey of Federated Recommender Systems: Techniques, Challenges, and Future Directionscited by 0
  17. 2026Resource-Adaptive Federated Text Generation with Differential Privacycited by 0
  18. 2026PubSwap: Public-Data Off-Policy Coordination for Federated RLVRcited by 0