Empirical Calibration and Metric Differential Privacy in Language Models
Not on the map. Why?
Abstract (arXiv)
NLP models trained with differential privacy (DP) usually adopt the DP-SGD framework, and privacy guarantees are often reported in terms of the privacy budget ε. However, ε does not have any intrinsic meaning, and it is generally not possible to compare across variants of the framework. Work in image processing has therefore explored how to empirically calibrate noise across frameworks using Membership Inference Attacks (MIAs). However, this kind of calibration has not been established for NLP. In this paper, we show that MIAs offer little help in calibrating privacy, whereas reconstruction attacks are more useful. As a use case, we define a novel kind of directional privacy based on the von Mises-Fisher (VMF) distribution, a metric DP mechanism that perturbs angular distance rather than adding (isotropic) Gaussian noise, and apply this to NLP architectures. We show that, even though formal guarantees are incomparable, empirical privacy calibration reveals that each mechanism has different areas of strength with respect to utility-privacy trade-offs.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 53 of its 91 references that are in the map. First posted on arXiv 18 Mar 2025.
Builds on
53 of 91 references.
References · 53
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
- 2016Deep Learning with Differential Privacycited by 2,682
- 2019Deep Leakage from Gradientscited by 1,004
- 2018Neural Network Acceptability Judgmentscited by 952
- 2020Inverting Gradients -- How easy is it to break privacy in federated learning?cited by 707
- 2017Renyi Differential Privacycited by 643
- 2021Large Language Models Can Be Strong Differentially Private Learnerscited by 405
- 2020Label-Only Membership Inference Attackscited by 371
- 2020iDLG: Improved Deep Leakage from Gradientscited by 367
- 2016Concentrated Differential Privacy: Simplifications, Extensions, and Lower Boundscited by 339
- 2021Membership Inference Attacks on Machine Learning: A Surveycited by 306
- 2023Membership Inference Attacks against Language Models via Neighbourhood Comparisoncited by 249
38 more
- 2021Enhanced Membership Inference Attacks against Machine Learning Modelscited by 237
- 2020Differentially Private Learning Needs Better Features (or Much More Data)cited by 233
- 2022Unlocking High-Accuracy Differentially Private Image Classification through Scalecited by 218
- 2020Auditing Differentially Private Machine Learning: How Private is Private SGD?cited by 213
- 2021Trustworthy AI: From Principles to Practicescited by 206
- 2024Do Membership Inference Attacks Work on Large Language Models?cited by 188
- 2022Quantifying Privacy Risks of Masked Language Models Using Membership Inference Attackscited by 168
- 2023How to DP-fy ML: A Practical Guide to Machine Learning with Differential Privacycited by 150
- 2016Concentrated Differential Privacycited by 139
- 2021On the Importance of Difficulty Calibration in Membership Inference Attackscited by 137
- 2021Large-Scale Differentially Private BERTcited by 117
- 2019Privacy- and Utility-Preserving Textual Analysis via Calibrated Multivariate Perturbationscited by 115
- 2021Robbing the Fed: Directly Obtaining Private Data in Federated Learning with Modified Modelscited by 106
- 2018Comprehensive Privacy Analysis of Deep Learning: Passive and Active White-box Inference Attacks against Centralized and Federated Learningcited by 75
- 2021Differential Privacy for Text Analytics via Natural Text Sanitizationcited by 74
- 2022Recovering Private Text in Federated Learning of Language Modelscited by 67
- 2020ML Privacy Meter: Aiding Regulatory Compliance by Quantifying the Privacy Risks of Machine Learningcited by 66
- 2021TAG: Gradient Attack on Transformer-based Language Modelscited by 63
- 2022The Limits of Word Level Differential Privacycited by 63
- 2018Generalised Differential Privacy for Text Document Processingcited by 52
- 2022Decepticons: Corrupted Transformers Breach Privacy in Federated Learning for Language Modelscited by 52
- 2022LAMP: Extracting Text from Gradients with Language Model Priorscited by 52
- 2019Leveraging Hierarchical Representations for Preserving Privacy and Utility in Textcited by 49
- 2020Differentially Private Language Models Benefit from Public Pre-trainingcited by 46
- 2021TEM: High Utility Metric Differential Privacy on Textcited by 43
- 2022Bounding Training Data Reconstruction in Private (Deep) Learningcited by 43
- 2023Bounding Training Data Reconstruction in DP-SGDcited by 42
- 2022Gradient Obfuscation Gives a False Sense of Security in Federated Learningcited by 37
- 2021ADePT: Auto-encoder based Differentially Private Text Transformationcited by 31
- 2021PRECODE - A Generic Model Extension to Prevent Deep Gradient Leakagecited by 28
- 2023DP-BART for Privatized Text Rewriting under Local Differential Privacycited by 27
- 2023Differentially Private Natural Language Models: Recent Advances and Future Directionscited by 20
- 2021Revealing and Protecting Labels in Distributed Trainingcited by 17
- 2024Reconstruct Your Previous Conversations! Comprehensively Investigating Privacy Leakage Risks in Conversations with GPT Modelscited by 16
- 2022Differential Privacy in Natural Language Processing: The Story So Farcited by 12
- 2023Recent Advances of Differential Privacy in Centralized Deep Learning: A Systematic Surveycited by 12
- 2016Whiteout: Gaussian Adaptive Noise Regularization in Deep Neural Networkscited by 8
- 2022Reconstruction Attack on Differential Private Trajectory Protection Mechanismscited by 7
50 most cited of 53 All 53 (JSON)
Built on by
Citing papers per year: 2026: 1.
1 paper.