mlsanityData to 7 Oct 2026

Large-Scale Differentially Private BERT

arXiv 2108.01624 ↗First posted 3 Aug 2021cs.LG cs.CL cs.CRcited by 117

Not on the map. Why?

Abstract (arXiv)

In this work, we study the large-scale pretraining of BERT-Large with differentially private SGD (DP-SGD). We show that combined with a careful implementation, scaling up the batch size to millions (i.e., mega-batches) improves the utility of the DP-SGD step for BERT; we also enhance its efficiency by using an increasing batch size schedule. Our implementation builds on the recent work of [SVK20], who demonstrated that the overhead of a DP-SGD step is minimized with effective use of JAX [BFH+18, FJL18] primitives in conjunction with the XLA compiler [XLA17]. Our implementation achieves a masked language model accuracy of 60.5% at a batch size of 2M, for ε = 5.36. To put this number in perspective, non-private BERT models achieve an accuracy of ~70%.

Key facts. Cited by 117 papers in the map (data through 7 Oct 2026). Builds on 41 of its 64 references that are in the map. First posted on arXiv 3 Aug 2021.

Builds on

41 of 64 references.

References · 41

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  5. 2016Layer Normalizationcited by 5,579
  6. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  7. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  8. 2016Deep Learning with Differential Privacycited by 2,682
  9. 2020Extracting Training Data from Large Language Modelscited by 2,144
  10. 2017Decoupled Weight Decay Regularizationcited by 1,722
  11. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  12. 2021MLP-Mixer: An all-MLP Architecture for Visioncited by 1,441
29 more
  1. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,041
  2. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 896
  3. 2018The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networkscited by 847
  4. 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 671
  5. 2017Renyi Differential Privacycited by 643
  6. 2017In-Datacenter Performance Analysis of a Tensor Processing Unitcited by 610
  7. 2016Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networkscited by 541
  8. 2019Does Learning Require Memorization? A Short Tale about a Long Tailcited by 452
  9. 2018Shampoo: Preconditioned Stochastic Tensor Optimizationcited by 413
  10. 2021FNet: Mixing Tokens with Fourier Transformscited by 343
  11. 2017Don't Decay the Learning Rate, Increase the Batch Sizecited by 313
  12. 2018Privacy Amplification by Subsampling: Tight Analyses via Couplings and Divergencescited by 237
  13. 2019Differentially Private Learning with Adaptive Clippingcited by 218
  14. 2018Measuring the Effects of Data Parallelism on Neural Network Trainingcited by 190
  15. 2021Numerical Composition of Differential Privacycited by 174
  16. 2018Large scale distributed neural network training through online distillationcited by 144
  17. 2021GSPMD: General and Scalable Parallelization for ML Computation Graphscited by 109
  18. 2020When is Memorization of Irrelevant Training Data Necessary for High-Accuracy Learning?cited by 102
  19. 2019AdaCliP: Adaptive Clipping for Private SGDcited by 77
  20. 2018Norm matters: efficient and accurate normalization schemes in deep networkscited by 62
  21. 2019Robust Bi-Tempered Logistic Loss Based on Bregman Divergencescited by 61
  22. 2020Enabling Fast Differentially Private SGD via Just-in-Time Compilation and Vectorizationcited by 58
  23. 2017Riemannian approach to batch normalizationcited by 35
  24. 2017Learning Differentially Private Recurrent Language Modelscited by 32
  25. 2021A Large Batch Optimizer Reality Check: Traditional, Generic Optimizers Suffice Across Batch Sizescited by 20
  26. 2019Memory-Efficient Adaptive Optimizationcited by 19
  27. 2019Faster Neural Network Training with Data Echoingcited by 19
  28. 2021LocoProp: Enhancing BackProp via Local Loss Optimizationcited by 15
  29. 2020Stochastic Optimization with Laggard Data Pipelinescited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 7; 2022: 29; 2023: 26; 2024: 34; 2025: 13; 2026: 8.

117 papers.

Sort:

By area:

  1. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  2. 2022Quantifying Memorization Across Neural Language Modelscited by 755
  3. 2022Knowledge Unlearning for Mitigating Privacy Risks in Language Modelscited by 424
  4. 2021Large Language Models Can Be Strong Differentially Private Learnerscited by 405
  5. 2021Differentially Private Fine-tuning of Language Modelscited by 392
  6. 2022Unlocking High-Accuracy Differentially Private Image Classification through Scalecited by 218
  7. 2023Unlearn What You Want to Forget: Efficient Unlearning for LLMscited by 211
  8. 2022Are Large Pre-Trained Language Models Leaking Your Personal Information?cited by 195
  9. 2022What Does it Mean for a Language Model to Preserve Privacy?cited by 194
  10. 2024Security and Privacy Challenges of Large Language Models: A Surveycited by 179
  11. 2023How to DP-fy ML: A Practical Guide to Machine Learning with Differential Privacycited by 150
  12. 2023Emergent and Predictable Memorization in Large Language Modelscited by 147
38 more
  1. 2022Differentially Private Diffusion Modelscited by 118
  2. 2023Foundation Models and Fair Usecited by 112
  3. 2022Preventing Verbatim Memorization in Language Models Gives a False Sense of Privacycited by 97
  4. 2022Toward Training at ImageNet Scale with Differential Privacycited by 87
  5. 2022Position: Considerations for Differentially Private Learning with Large-Scale Public Pretrainingcited by 86
  6. 2023Flocks of Stochastic Parrots: Differentially Private Prompt Learning for Large Language Modelscited by 80
  7. 2023The MiniPile Challenge for Data-Efficient Language Modelscited by 74
  8. 2024Differentially Private Synthetic Data via Foundation Model APIs 2: Textcited by 74
  9. 2022Differentially Private Language Models for Secure Data Sharingcited by 68
  10. 2021Differential privacy and robust statistics in high dimensionscited by 64
  11. 2023Differentially Private Synthetic Data via Foundation Model APIs 1: Imagescited by 63
  12. 2022Exploring the Limits of Differentially Private Deep Learning with Group-wise Clippingcited by 52
  13. 2022A Customized Text Sanitization Mechanism with Differential Privacycited by 51
  14. 2022TAN Without a Burn: Scaling Laws of DP-SGDcited by 51
  15. 2024Be like a Goldfish, Don't Memorize! Mitigating Memorization in Generative LLMscited by 50
  16. 2024Operationalizing a Threat Model for Red-Teaming Large Language Models (LLMs)cited by 44
  17. 2021Node-Level Differentially Private Graph Neural Networkscited by 43
  18. 2023Privacy-Preserving Parameter-Efficient Fine-Tuning for Large Language Model Servicescited by 43
  19. 2024To Each (Textual Sequence) Its Own: Improving Memorized-Data Unlearning in Large Language Modelscited by 42
  20. 2023Differentially Private Low-Rank Adaptation of Large Language Model Using Federated Learningcited by 41
  21. 2022Large Scale Transfer Learning for Differentially Private Image Classificationcited by 40
  22. 2023One-shot Empirical Privacy Estimation for Federated Learningcited by 35
  23. 2024Privacy-Preserving Instructions for Aligning Large Language Modelscited by 34
  24. 2022Just Fine-tune Twice: Selective Differential Privacy for Large Language Modelscited by 32
  25. 2024On the Privacy Risk of In-context Learningcited by 31
  26. 2023Identifying and Mitigating Privacy Risks Stemming from Language Models: A Surveycited by 30
  27. 2024Privacy Leakage on DNNs: A Survey of Model Inversion Attacks and Defensescited by 30
  28. 2024PrE-Text: Training Language Models on Private Federated Data in the Age of LLMscited by 29
  29. 2022On the Impossible Safety of Large AI Modelscited by 27
  30. 2023Quantifying Association Capabilities of Large Language Models and Its Implications on Privacy Leakagecited by 26
  31. 2024Learnable Privacy Neurons Localization in Language Modelscited by 26
  32. 2023MoPe: Model Perturbation-based Privacy Attacks on Language Modelscited by 25
  33. 2022Learning to Generate Image Embeddings with User-level Differential Privacycited by 23
  34. 2024Deconstructing The Ethics of Large Language Models from Long-standing Issues to New-emerging Dilemmas: A Surveycited by 23
  35. 2023Selective Pre-training for Private Fine-tuningcited by 22
  36. 2024How Private are DP-SGD Implementations?cited by 22
  37. 2025Privacy Auditing of Large Language Modelscited by 22
  38. 2024Mind the Privacy Unit! User-Level Differential Privacy for Language Model Fine-Tuningcited by 21

50 most cited of 117 All 117 (JSON)