mlsanityData to 14 Oct 2026

Natural Language Understanding with Privacy-Preserving BERT

arXiv 2104.07504 ↗First posted 15 Apr 2021cs.CLcited by 55

Not on the map. Why?

Abstract (arXiv)

Privacy preservation remains a key challenge in data mining and Natural Language Understanding (NLU). Previous research shows that the input text or even text embeddings can leak private information. This concern motivates our research on effective privacy preservation approaches for pretrained Language Models (LMs). We investigate the privacy and utility implications of applying dx-privacy, a variant of Local Differential Privacy, to BERT fine-tuning in NLU applications. More importantly, we further propose privacy-adaptive LM pretraining methods and show that our approach can boost the utility of BERT dramatically while retaining the same level of privacy protection. We also quantify the level of privacy preservation and provide guidance on privacy configuration. Our experiments and findings lay the groundwork for future explorations of privacy-preserving NLU with pretrained LMs.

Key facts. Cited by 55 papers in the map (data through 14 Oct 2026). Builds on 24 of its 41 references that are in the map. First posted on arXiv 15 Apr 2021.

Builds on

24 of 41 references.

References · 24

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  4. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  5. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  6. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,474
  7. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,811
  8. 2016Deep Learning with Differential Privacycited by 2,728
  9. 2015Character-level Convolutional Networks for Text Classificationcited by 2,330
  10. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,475
  11. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,043
  12. 2018Certified Robustness to Adversarial Examples with Differential Privacycited by 468
12 more
  1. 2020Information Leakage in Embedding Modelscited by 203
  2. 2020Threats to Federated Learning: A Surveycited by 178
  3. 2018Protection Against Reconstruction and Its Applications in Private Federated Learningcited by 130
  4. 2019Privacy- and Utility-Preserving Textual Analysis via Calibrated Multivariate Perturbationscited by 119
  5. 2018Towards Robust and Privacy-preserving Text Representationscited by 75
  6. 2019BERT with History Answer Embedding for Conversational Question Answeringcited by 67
  7. 2020Differentially Private Representation for NLP: Formal Guarantee and An Empirical Study on Privacy and Fairnesscited by 67
  8. 2018Privacy-preserving Neural Representations of Textcited by 54
  9. 2018Generalised Differential Privacy for Text Document Processingcited by 52
  10. 2020Towards Differentially Private Text Representationscited by 29
  11. 2018SynTF: Synthetic and Differentially Private Term Frequency Vectors for Privacy-Preserving Text Miningcited by 21
  12. 2019Generic Intent Representation in Web Searchcited by 12

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 9; 2023: 14; 2024: 15; 2025: 10; 2026: 7.

55 papers.

Sort:

By area:

  1. 2024Improving LoRA in Privacy-preserving Federated Learningcited by 161
  2. 2023How to DP-fy ML: A Practical Guide to Machine Learning with Differential Privacycited by 153
  3. 2022Synthetic Text Generation with Differential Privacy: A Simple and Practical Recipecited by 108
  4. 2022When Federated Learning Meets Pre-trained Language Models' Parameter-Efficient Tuning Methodscited by 75
  5. 2023DP-Forward: Fine-tuning and Inference on Language Models with Differential Privacy in Forward Passcited by 67
  6. 2022THE-X: Privacy-Preserving Transformer Inference with Homomorphic Encryptioncited by 57
  7. 2022A Customized Text Sanitization Mechanism with Differential Privacycited by 52
  8. 2023Privacy in Large Language Models: Attacks, Defenses and Future Directionscited by 51
  9. 2023Privacy-Preserving Parameter-Efficient Fine-Tuning for Large Language Model Servicescited by 43
  10. 2023PrivLM-Bench: A Multi-level Privacy Evaluation Benchmark for Language Modelscited by 37
  11. 2023Split-and-Denoise: Protect large language model inference with local differential privacycited by 32
  12. 2023Identifying and Mitigating Privacy Risks Stemming from Language Models: A Surveycited by 30
38 more
  1. 2024PrivacyRestore: Privacy-Preserving Inference in Large Language Models via Privacy Removal and Restorationcited by 28
  2. 2023Grounding Foundation Models through Federated Transfer Learning: A General Frameworkcited by 24
  3. 2023Differentially Private Natural Language Models: Recent Advances and Future Directionscited by 20
  4. 2025Can Language Models Follow Multiple Turns of Entangled Instructions?cited by 14
  5. 2024No Free Lunch Theorem for Privacy-Preserving LLM Inferencecited by 13
  6. 2024The Fire Thief Is Also the Keeper: Balancing Usability and Privacy in Promptscited by 13
  7. 2023A Split-and-Privatize Framework for Large Language Model Fine-Tuningcited by 11
  8. 2025FedSVD: Adaptive Orthogonalization for Private Federated Learning with LoRAcited by 11
  9. 2022Privacy-Preserving Text Classification on BERT Embeddings with Homomorphic Encryptioncited by 9
  10. 2023SentinelLMs: Encrypted Input Adaptation and Fine-tuning of Language Models for Private and Secure Inferencecited by 9
  11. 2024EmojiPrompt: Generative Prompt Obfuscation for Privacy-Preserving Communication with Cloud-based LLMscited by 9
  12. 2024More RLHF, More Trust? On The Impact of Preference Alignment On Trustworthinesscited by 9
  13. 2024Privacy-Preserving Language Model Inference with Instance Obfuscationcited by 7
  14. 2022When Fairness Meets Privacy: Fair Classification with Semi-Private Sensitive Attributescited by 6
  15. 2025Towards Harnessing the Collaborative Power of Large and Small Models for Domain Taskscited by 4
  16. 2023ConfusionPrompt: Practical Private Inference for Online Large Language Modelscited by 3
  17. 2024Efficient and Scalable Implementation of Differentially Private Deep Learning without Shortcutscited by 3
  18. 2024Private Language Models via Truncated Laplacian Mechanismcited by 3
  19. 2023LatticeGen: A Cooperative Framework which Hides Generated Text in a Lattice for Privacy-Aware Generation on Cloudcited by 2
  20. 2024Federated Domain-Specific Knowledge Transfer on Large Language Models Using Synthetic Datacited by 2
  21. 2025Enhancing Privacy in the Early Detection of Sexual Predators Through Federated Learning and Differential Privacycited by 2
  22. 2025Anti-adversarial Learning: Desensitizing Prompts for Large Language Modelscited by 2
  23. 2025Operator-Theoretic Framework for Gradient-Free Federated Learningcited by 2
  24. 2022Directional Privacy for Deep Learningcited by 1
  25. 2022Swing Distillation: A Privacy-Preserving Knowledge Distillation Frameworkcited by 1
  26. 2024Protecting Privacy in Classifiers by Token Manipulationcited by 1
  27. 2025PrivTune: Efficient and Privacy-Preserving Fine-Tuning of Large Language Models via Device-Cloud Collaborationcited by 1
  28. 2026NOIR: Privacy-Preserving Generation of Code with Open-Source LLMscited by 1
  29. 2026Towards Privacy-Preserving Machine Translation at the Inference Stage: A New Task and Benchmarkcited by 1
  30. 2026SurrogateShield: Beyond Redaction for High-Utility, Privacy-Preserving LLM Interactionscited by 1
  31. 2022Production of Categorical Data Verifying Differential Privacy: Conception and Applications to Machine Learningcited by 0
  32. 2023Disentangling the Linguistic Competence of Privacy-Preserving BERTcited by 0
  33. 2024Characterizing Stereotypical Bias from Privacy-preserving Pre-Trainingcited by 0
  34. 2024FedMCP: Parameter-Efficient Federated Learning with Model-Contrastive Personalizationcited by 0
  35. 2024A Different Level Text Protection Mechanism With Differential Privacycited by 0
  36. 2024Preempting Text Sanitization Utility in Resource-Constrained Privacy-Preserving LLM Interactionscited by 0
  37. 2025Token-Level Privacy in Large Language Modelscited by 0
  38. 2025Theoretically Unmasking Inference Attacks Against LDP-Protected Clients in Federated Vision Modelscited by 0

50 most cited of 55 All 55 (JSON)