mlsanityData to 14 Oct 2026

Sparsity-Preserving Differentially Private Training of Large Embedding Models

arXiv 2311.08357 ↗First posted 14 Nov 2023cs.LG cs.CRcited by 5

Not on the map. Why?

Abstract (arXiv)

As the use of large embedding models in recommendation systems and language applications increases, concerns over user data privacy have also risen. DP-SGD, a training algorithm that combines differential privacy with stochastic gradient descent, has been the workhorse in protecting user privacy without compromising model accuracy by much. However, applying DP-SGD naively to embedding models can destroy gradient sparsity, leading to reduced training efficiency. To address this issue, we present two new algorithms, DP-FEST and DP-AdaFEST, that preserve gradient sparsity during private training of large embedding models. Our algorithms achieve substantial reductions (106 ×) in gradient size, while maintaining comparable levels of accuracy, on benchmark real-world datasets.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 43 of its 58 references that are in the map. First posted on arXiv 14 Nov 2023.

Builds on

43 of 58 references.

References · 43

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  4. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  5. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,375
  6. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  7. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  8. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  9. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,474
  10. 2016Deep Learning with Differential Privacycited by 2,728
  11. 2020Extracting Training Data from Large Language Modelscited by 2,189
  12. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 915
31 more
  1. 2016Wide & Deep Learning for Recommender Systemscited by 747
  2. 2017DeepFM: A Factorization-Machine based Neural Network for CTR Predictioncited by 676
  3. 2017Deep Interest Network for Click-Through Rate Predictioncited by 544
  4. 2021Large Language Models Can Be Strong Differentially Private Learnerscited by 412
  5. 2021Differentially Private Fine-tuning of Language Modelscited by 400
  6. 2017Deep & Cross Network for Ad Click Predictionscited by 393
  7. 2019Deep Learning Recommendation Model for Personalization and Recommendation Systemscited by 327
  8. 2020DCN V2: Improved Deep & Cross Network and Practical Lessons for Web-scale Learning to Rank Systemscited by 327
  9. 2020Differentially Private Learning Needs Better Features (or Much More Data)cited by 235
  10. 2022Unlocking High-Accuracy Differentially Private Image Classification through Scalecited by 224
  11. 2023TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning with Hardware Support for Embeddingscited by 202
  12. 2022Are Large Pre-Trained Language Models Leaking Your Personal Information?cited by 200
  13. 2021Numerical Composition of Differential Privacycited by 180
  14. 2023How to DP-fy ML: A Practical Guide to Machine Learning with Differential Privacycited by 153
  15. 2020Tempered Sigmoid Activations for Deep Learning with Differential Privacycited by 122
  16. 2022Toward Training at ImageNet Scale with Differential Privacycited by 87
  17. 2021Optimal Accounting of Differential Privacy via Characteristic Functioncited by 86
  18. 2021Software-Hardware Co-design for Fast and Scalable Training of Deep Learning Recommendation Modelscited by 78
  19. 2022Recovering Private Text in Federated Learning of Language Modelscited by 70
  20. 2020Distributed Hierarchical GPU Parameter Server for Massive Scale Deep Learning Ads Systemscited by 60
  21. 2022Connect the Dots: Tighter Discrete Approximations of Privacy Loss Distributionscited by 52
  22. 2021Not all noise is accounted equally: How differentially private learning benefits from large sampling ratescited by 21
  23. 2022Merlin HugeCTR: GPU-accelerated Recommender System Training and Inferencecited by 20
  24. 2021Wide Network Learning with Differential Privacycited by 17
  25. 2022Faster Privacy Accounting via Evolving Discretizationcited by 15
  26. 2020Differentially Private Set Unioncited by 14
  27. 2020Differentially private partition selectioncited by 6
  28. 2022Private Ad Modeling with DP-SGDcited by 6
  29. 2020Differentially Private Weighted Samplingcited by 3
  30. 2018The Graph-Based Behavior-Aware Recommendation for Interactive Newscited by 1
  31. 2023DP-SIPS: A simpler, more scalable mechanism for differentially private partition selectioncited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 1; 2026: 2.

5 papers.

  1. 2024AI Risk Management Should Incorporate Both Safety and Securitycited by 17
  2. 2026JAX-Privacy: A library for differentially private machine learningcited by 3
  3. 2024Differentially Private Optimization with Sparse Gradientscited by 2
  4. 2025Cocoon: A System Architecture for Differentially Private Training with Correlated Noisescited by 1
  5. 2026When Do Fewer Coordinates Suffice in DP-SGD?cited by 0