mlsanityData to 14 Oct 2026

Selective Pre-training for Private Fine-tuning

arXiv 2305.13865 ↗First posted 23 May 2023cs.LG cs.CRcited by 23

Not on the map. Why?

Abstract (arXiv)

Text prediction models, when used in applications like email clients or word processors, must protect user data privacy and adhere to model size constraints. These constraints are crucial to meet memory and inference time requirements, as well as to reduce inference costs. Building small, fast, and private domain-specific language models is a thriving area of research. In this work, we show that a careful pre-training on a subset of the public dataset that is guided by the private dataset is crucial to train small language models with differential privacy. On standard benchmarks, small models trained with our new framework achieve state-of-the-art performance. In addition to performance improvements, our results demonstrate that smaller models, through careful pre-training and private fine-tuning, can match the performance of much larger models that do not have access to private data. This underscores the potential of private learning for model compression and enhanced efficiency.

Key facts. Cited by 23 papers in the map (data through 14 Oct 2026). Builds on 74 of its 101 references that are in the map. First posted on arXiv 23 May 2023.

Builds on

74 of 101 references.

References · 74

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  5. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  6. 2020Scaling Laws for Neural Language Modelscited by 6,586
  7. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  8. 2017On Calibration of Modern Neural Networkscited by 4,960
  9. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  10. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  11. 2016Deep Learning with Differential Privacycited by 2,728
  12. 2016Membership Inference Attacks against Machine Learning Modelscited by 2,281
38 more
  1. 2020Extracting Training Data from Large Language Modelscited by 2,189
  2. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 2,031
  3. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,565
  4. 2022Quantifying Memorization Across Neural Language Modelscited by 763
  5. 2021Deduplicating Training Data Makes Language Models Bettercited by 714
  6. 2022Beyond neural scaling laws: beating power law scaling via data pruningcited by 528
  7. 2021Large Language Models Can Be Strong Differentially Private Learnerscited by 412
  8. 2021Differentially Private Fine-tuning of Language Modelscited by 400
  9. 2020Label-Only Membership Inference Attackscited by 388
  10. 2016Semi-supervised Knowledge Transfer for Deep Learning from Private Training Datacited by 348
  11. 2019Selection via Proxy: Efficient Data Selection for Deep Learningcited by 339
  12. 2023Data Selection for Language Models via Importance Resamplingcited by 329
  13. 2023SemDeDup: Data-efficient learning at web-scale through semantic deduplicationcited by 303
  14. 2018Scalable Private Learning with PATEcited by 269
  15. 2020Differentially Private Learning Needs Better Features (or Much More Data)cited by 235
  16. 2022Unlocking High-Accuracy Differentially Private Image Classification through Scalecited by 224
  17. 2021Numerical Composition of Differential Privacycited by 180
  18. 2022Structured Pruning Learns Compact and Accurate Modelscited by 170
  19. 2021Counterfactual Memorization in Neural Language Modelscited by 156
  20. 2022Scaling Laws and Interpretability of Learning from Repeated Datacited by 152
  21. 2021Large-Scale Differentially Private BERTcited by 118
  22. 2014Differentially Private Empirical Risk Minimization: Efficient Algorithms and Tight Error Boundscited by 107
  23. 2021Hyperparameter Tuning with Renyi Differential Privacycited by 104
  24. 2020Bypassing the Ambient Dimension: Private SGD with Gradient Subspace Identificationcited by 90
  25. 2021Large Scale Private Learning via Low-rank Reparametrizationcited by 88
  26. 2022Position: Considerations for Differentially Private Learning with Large-Scale Public Pretrainingcited by 88
  27. 2022Toward Training at ImageNet Scale with Differential Privacycited by 87
  28. 2023Membership Inference Attacks against Diffusion Modelscited by 86
  29. 2018Private Selection from Private Candidatescited by 75
  30. 2020Enabling Fast Differentially Private SGD via Just-in-Time Compilation and Vectorizationcited by 58
  31. 2022Mixed Differential Privacy in Computer Visioncited by 54
  32. 2019Transfer Adaptation Learning: A Decade Surveycited by 52
  33. 2022TAN Without a Burn: Scaling Laws of DP-SGDcited by 52
  34. 2022Exploring the Limits of Differentially Private Deep Learning with Group-wise Clippingcited by 52
  35. 2021Public Data-Assisted Mirror Descent for Private Model Trainingcited by 49
  36. 2022Scalable and Efficient Training of Large Convolutional Neural Networks with Differential Privacycited by 48
  37. 2022When Does Differentially Private Learning Not Suffer in High Dimensions?cited by 48
  38. 2022Differentially Private Bias-Term Fine-tuning of Foundation Modelscited by 48

50 most cited of 74 All 74 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 4; 2024: 10; 2025: 6; 2026: 3.

23 papers.

Sort:

By area:

  1. 2023Textbooks Are All You Needcited by 569
  2. 2023Privacy in Large Language Models: Attacks, Defenses and Future Directionscited by 51
  3. 2024LLM-PBE: Assessing Data Privacy in Large Language Modelscited by 41
  4. 2024What is the Role of Small Models in the LLM Era: A Surveycited by 41
  5. 2024PrivacyRestore: Privacy-Preserving Inference in Large Language Models via Privacy Removal and Restorationcited by 28
  6. 2023Grounding Foundation Models through Federated Transfer Learning: A General Frameworkcited by 24
  7. 2025Ten Challenging Problems in Federated Foundation Modelscited by 20
  8. 2024Differentially Private Knowledge Distillation via Synthetic Text Generationcited by 18
  9. 2024Prompt Public Large Language Models to Synthesize Data for Private On-device Applicationscited by 17
  10. 2025Differentially Private Synthetic Data via APIs 3: Using Simulators Instead of Foundation Modelcited by 13
  11. 2025DPImageBench: A Unified Benchmark for Differentially Private Image Synthesiscited by 11
  12. 2024Efficiently Computing Similarities to Private Datasetscited by 9
11 more
  1. 2024Towards More Realistic Extraction Attacks: An Adversarial Perspectivecited by 8
  2. 2023Optimal Locally Private Nonparametric Classification with Public Datacited by 7
  3. 2025AugFL: Augmenting Federated Learning with Pretrained Modelscited by 6
  4. 2024Small Language Models for Application Interactions: A Case Studycited by 5
  5. 2024Differentially Private Kernel Density Estimationcited by 3
  6. 2024Federated Domain-Specific Knowledge Transfer on Large Language Models Using Synthetic Datacited by 2
  7. 2025Public Data Assisted Differentially Private In-Context Learningcited by 1
  8. 2025Towards Reliable and Generalizable Differentially Private Machine Learning (Extended Version)cited by 0
  9. 2026Curation Leaks: Membership Inference Attacks against Data Curation for Machine Learningcited by 0
  10. 2026Privacy-Preserving RAG via Multi-Agent Semantic Rewriting: Achieving Confidentiality Without Compromising Contextual Fidelitycited by 0
  11. 2026Differentially Private Mixing of Public Datasets Improves Private Learningcited by 0