mlsanityData to 7 Oct 2026

Differentially Private Model Compression

arXiv 2206.01838 ↗First posted 3 Jun 2022cs.LG cs.CRcited by 13

Not on the map. Why?

Abstract (arXiv)

Recent papers have shown that large pre-trained language models (LLMs) such as BERT, GPT-2 can be fine-tuned on private data to achieve performance comparable to non-private models for many downstream Natural Language Processing (NLP) tasks while simultaneously guaranteeing differential privacy. The inference cost of these models -- which consist of hundreds of millions of parameters -- however, can be prohibitively large. Hence, often in practice, LLMs are compressed before they are deployed in specific applications. In this paper, we initiate the study of differentially private model compression and propose frameworks for achieving 50% sparsity levels while maintaining nearly full performance. We demonstrate these ideas on standard GLUE benchmarks using BERT models, setting benchmarks for future research on this topic.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Builds on 56 of its 84 references that are in the map. First posted on arXiv 3 Jun 2022.

Builds on

56 of 84 references.

References · 56

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  6. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  7. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  8. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  9. 2016Deep Learning with Differential Privacycited by 2,682
  10. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
  11. 2016Membership Inference Attacks against Machine Learning Modelscited by 2,221
  12. 2020Extracting Training Data from Large Language Modelscited by 2,144
38 more
  1. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 1,998
  2. 2014FitNets: Hints for Thin Deep Netscited by 1,619
  3. 2020Knowledge Distillation: A Surveycited by 1,578
  4. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,153
  5. 2018MnasNet: Platform-Aware Neural Architecture Search for Mobilecited by 937
  6. 2018The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networkscited by 847
  7. 2021Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networkscited by 521
  8. 2019Patient Knowledge Distillation for BERT Model Compressioncited by 502
  9. 2021Large Language Models Can Be Strong Differentially Private Learnerscited by 405
  10. 2021Differentially Private Fine-tuning of Language Modelscited by 392
  11. 2016Semi-supervised Knowledge Transfer for Deep Learning from Private Training Datacited by 347
  12. 2019Differential Privacy Has Disparate Impact on Model Accuracycited by 316
  13. 2019Q8BERT: Quantized 8Bit BERTcited by 296
  14. 2020The Lottery Ticket Hypothesis for Pre-trained BERT Networkscited by 275
  15. 2018Scalable Private Learning with PATEcited by 266
  16. 2021Pruning and Quantization for Deep Neural Network Acceleration: A Surveycited by 259
  17. 2020Differentially Private Learning Needs Better Features (or Much More Data)cited by 233
  18. 2022Unlocking High-Accuracy Differentially Private Image Classification through Scalecited by 218
  19. 2021Numerical Composition of Differential Privacycited by 174
  20. 2021Efficient Deep Learning: A Survey on Making Deep Learning Models Smaller, Faster, and Bettercited by 153
  21. 2019Distilling Task-Specific Knowledge from BERT into Simple Neural Networkscited by 152
  22. 2020Training with Quantization Noise for Extreme Model Compressioncited by 145
  23. 2019Well-Read Students Learn Better: On the Importance of Pre-training Compact Modelscited by 118
  24. 2021Large-Scale Differentially Private BERTcited by 117
  25. 2020Compressing Large-Scale Transformer-Based Models: A Case Study on BERTcited by 95
  26. 2021Large Scale Private Learning via Low-rank Reparametrizationcited by 87
  27. 2021Do Not Let Privacy Overbill Utility: Gradient Embedding Perturbation for Private Learningcited by 86
  28. 2020Privacy in Deep Learning: A Surveycited by 70
  29. 2020EdgeBERT: Sentence-Level Energy Optimizations for Latency-Aware Multi-Task NLP Inferencecited by 69
  30. 2021NAS-BERT: Task-Agnostic and Adaptive-Size BERT Compression with Neural Architecture Searchcited by 55
  31. 2020LadaBERT: Lightweight Adaptation of BERT through Hybrid Model Compressioncited by 45
  32. 2020Federated Model Distillation with Noise-Free Differential Privacycited by 40
  33. 2022Large Scale Transfer Learning for Differentially Private Image Classificationcited by 40
  34. 2020Efficient Transformer-based Large Scale Language Representations using Hardware-friendly Block Structured Pruningcited by 39
  35. 2020XtremeDistil: Multi-stage Distillation for Massive Multilingual Modelscited by 38
  36. 2018Private Model Compression via Knowledge Distillationcited by 37
  37. 2022Differentially Private Decoding in Large Language Modelscited by 34
  38. 2021DP-SGD vs PATE: Which Has Less Disparate Impact on Model Accuracy?cited by 32

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 5; 2024: 4; 2025: 3; 2026: 1.

13 papers.

Sort:

By area:

  1. 2023Membership Inference Attacks against Language Models via Neighbourhood Comparisoncited by 249
  2. 2024Safeguarding Large Language Models: A Surveycited by 73
  3. 2024Building Guardrails for Large Language Modelscited by 61
  4. 2023Privacy in Large Language Models: Attacks, Defenses and Future Directionscited by 48
  5. 2023Students Parrot Their Teachers: Membership Inference on Model Distillationcited by 31
  6. 2023Identifying and Mitigating Privacy Risks Stemming from Language Models: A Surveycited by 30
  7. 2023Selective Pre-training for Private Fine-tuningcited by 22
  8. 2024Differentially Private Zeroth-Order Methods for Scalable Large Language Model Finetuningcited by 14
  9. 2025AugFL: Augmenting Federated Learning with Pretrained Modelscited by 6
  10. 2026DP-OPD: Differentially Private On-Policy Distillation for Language Modelscited by 1
  11. 2024TensorTEE: Unifying Heterogeneous TEE Granularity for Efficient Secure Collaborative Tensor Computingcited by 0
  12. 2025Towards Reliable and Generalizable Differentially Private Machine Learning (Extended Version)cited by 0
1 more
  1. 2025Balancing Utility and Privacy: Dynamically Private SGD with Random Projectioncited by 0