mlsanityData to 7 Oct 2026

Benchmarking Differential Privacy and Federated Learning for BERT Models

arXiv 2106.13973 ↗First posted 26 Jun 2021cs.CL cs.CR cs.LGcited by 28

Not on the map. Why?

Abstract (arXiv)

Natural Language Processing (NLP) techniques can be applied to help with the diagnosis of medical conditions such as depression, using a collection of a person's utterances. Depression is a serious medical illness that can have adverse effects on how one feels, thinks, and acts, which can lead to emotional and physical problems. Due to the sensitive nature of such data, privacy measures need to be taken for handling and training models with such data. In this work, we study the effects that the application of Differential Privacy (DP) has, in both a centralized and a Federated Learning (FL) setup, on training contextualized language models (BERT, ALBERT, RoBERTa and DistilBERT). We offer insights on how to privately train NLP models and what architectures and setups provide more desirable privacy utility trade-offs. We envisage this work to be used in future healthcare and mental health studies to keep medical history private. Therefore, we provide an open-source implementation of this work.

Key facts. Cited by 28 papers in the map (data through 7 Oct 2026). Builds on 21 of its 47 references that are in the map. First posted on arXiv 26 Jun 2021.

Builds on

21 of 47 references.

References · 21

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  3. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  4. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  5. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,462
  6. 2019Advances and Open Problems in Federated Learningcited by 3,021
  7. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  8. 2016Deep Learning with Differential Privacycited by 2,682
  9. 2018Federated Learning with Non-IID Datacited by 1,099
  10. 2017Federated Multi-Task Learningcited by 669
  11. 2017Differentially Private Federated Learning: A Client Level Perspectivecited by 451
  12. 2019Federated Machine Learning: Concept and Applicationscited by 409
9 more
  1. 2019A Survey on Federated Learning Systems: Vision, Hype and Reality for Data Privacy and Protectioncited by 371
  2. 2020Differentially Private Learning Needs Better Features (or Much More Data)cited by 233
  3. 2018Federated Meta-Learning with Fast Convergence and Efficient Communicationcited by 154
  4. 2018Towards Robust and Privacy-preserving Text Representationscited by 75
  5. 2019Differentially Private Model Publishing for Deep Learningcited by 69
  6. 2014Private Learning and Sanitization: Pure vs. Approximate Differential Privacycited by 62
  7. 2018Parameter Sharing Methods for Multilingual Self-Attentional Translation Modelscited by 33
  8. 2021Privacy Regularization: Joint Privacy-Utility Optimization in Language Modelscited by 29
  9. 2020Benchmarking Differentially Private Residual Networks for Medical Imagerycited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 4; 2022: 9; 2023: 8; 2024: 3; 2025: 2; 2026: 2.

28 papers.

Sort:

By area:

  1. 2021Differentially Private Fine-tuning of Language Modelscited by 392
  2. 2022FedPrompt: Communication-Efficient and Privacy Preserving Prompt Tuning in Federated Learningcited by 86
  3. 2023The MiniPile Challenge for Data-Efficient Language Modelscited by 74
  4. 2022THE-X: Privacy-Preserving Transformer Inference with Homomorphic Encryptioncited by 55
  5. 2022Exploring the Limits of Differentially Private Deep Learning with Group-wise Clippingcited by 52
  6. 2021Just What do You Think You're Doing, Dave?' A Checklist for Responsible Data Use in NLPcited by 47
  7. 2022Memorization in NLP Fine-tuning Methodscited by 43
  8. 2023Identifying and Mitigating Privacy Risks Stemming from Language Models: A Surveycited by 30
  9. 2021UserIdentifier: Implicit User Representations for Simple and Effective Personalized Sentiment Analysiscited by 28
  10. 2023FwdLLM: Efficient FedLLM using Forward Gradientcited by 25
  11. 2022FedAdapter: Efficient Federated Learning for Modern NLPcited by 24
  12. 2021Privacy enabled Financial Text Classification using Differential Privacy and Federated Learningcited by 18
16 more
  1. 2023Are Chatbots Ready for Privacy-Sensitive Applications? An Investigation into Input Regurgitation and Prompt-Induced Sanitizationcited by 17
  2. 2022Federated Few-Shot Learning for Mobile NLPcited by 15
  3. 2022Differentially Private Graph Classification with GNNscited by 12
  4. 2022Differential Privacy in Natural Language Processing: The Story So Farcited by 12
  5. 2023On the Efficacy of Differentially Private Few-shot Image Classificationcited by 12
  6. 2023Safeguarding Data in Multimodal AI: A Differentially Private Approach to CLIP Trainingcited by 7
  7. 2025A Comprehensive Review of Datasets for Clinical Mental Health AI Systemscited by 3
  8. 2025Towards Privacy-aware Mental Health AI Models: Advances, Challenges, and Opportunitiescited by 2
  9. 2023A Federated Approach for Hate Speech Detectioncited by 1
  10. 2023Privacy Aware Question-Answering System for Online Mental Health Risk Assessmentcited by 1
  11. 2024Federated Learning in Healthcare: Model Misconducts, Security, Challenges, Applications, and Future Research Directions -- A Systematic Reviewcited by 1
  12. 2024Privately Learning from Graphs with Applications in Fine-tuning Large Language Modelscited by 1
  13. 2022Interpretability of Fine-grained Classification of Sadness and Depressioncited by 0
  14. 2024Which PPML Would a User Choose? A Structured Decision Support Framework for Developers to Rank PPML Techniques Based on User Acceptance Criteriacited by 0
  15. 2026Privacy Preserving Topic-wise Sentiment Analysis of the Iran Israel USA Conflict Using Federated Transformer Modelscited by 0
  16. 2026FedMental: Evaluating Federated Learning for Mental Health Detection from Social Media Datacited by 0