mlsanityData to 14 Oct 2026

How do we measure privacy in text? A survey of text anonymization metrics

arXiv 2512.01109 ↗First posted 30 Nov 2025cs.CLcited by 5

Not on the map. Why?

Abstract (arXiv)

In this work, we aim to clarify and reconcile metrics for evaluating privacy protection in text through a systematic survey. Although text anonymization is essential for enabling NLP research and model development in domains with sensitive data, evaluating whether anonymization methods sufficiently protect privacy remains an open challenge. In manually reviewing 47 papers that report privacy metrics, we identify and compare six distinct privacy notions, and analyze how the associated metrics capture different aspects of privacy risk. We then assess how well these notions align with legal privacy standards (HIPAA and GDPR), as well as user-centered expectations grounded in HCI studies. Our analysis offers practical guidance on navigating the landscape of privacy evaluation approaches further and highlights gaps in current practices. Ultimately, we aim to facilitate more robust, comparable, and legally aware privacy evaluations in text anonymization.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 31 of its 65 references that are in the map. First posted on arXiv 30 Nov 2025.

Builds on

31 of 65 references.

References · 31

  1. 2016Membership Inference Attacks against Machine Learning Modelscited by 2,281
  2. 2021Membership Inference Attacks From First Principlescited by 878
  3. 2018The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networkscited by 864
  4. 2023Can LLMs Keep a Secret? Testing Privacy Implications of Language Models via Contextual Integrity Theorycited by 230
  5. 2022Synthetic Text Generation with Differential Privacy: A Simple and Practical Recipecited by 108
  6. 2022The Text Anonymization Benchmark (TAB): A Dedicated Corpus and Evaluation Framework for Text Anonymizationcited by 95
  7. 2021Differential Privacy for Text Analytics via Natural Text Sanitizationcited by 77
  8. 2022The Limits of Word Level Differential Privacycited by 63
  9. 2022A Customized Text Sanitization Mechanism with Differential Privacycited by 52
  10. 2022SoK: Let the Privacy Games Begin! A Unified Treatment of Data Inference Privacy in Machine Learningcited by 51
  11. 2023Reducing Privacy Risks in Online Self-Disclosures with Language Modelscited by 47
  12. 2024Membership Inference Attacks Against In-Context Learningcited by 43
19 more
  1. 2023PrivLM-Bench: A Multi-level Privacy Evaluation Benchmark for Language Modelscited by 37
  2. 2024CLEAR: Towards Contextual LLM-Empowered Privacy Policy Analysis and Risk Generation for Large Language Model Applicationscited by 33
  3. 2024DP-MLM: Differentially Private Text Rewriting Using Masked Language Modelscited by 24
  4. 2024IncogniText: Privacy-enhancing Conditional Text Anonymization via LLM-based Private Attribute Randomizationcited by 23
  5. 2022PromptEHR: Conditional Electronic Healthcare Records Generation with Prompt Learningcited by 22
  6. 2024Rescriber: Smaller-LLM-Powered User-Led Data Minimization for LLM-Based Chatbotscited by 16
  7. 2025A False Sense of Privacy: Evaluating Textual Data Sanitization Beyond Surface-level Privacy Leakagecited by 16
  8. 2023TMI! Finetuned Models Leak Private Information from their Pretraining Datacited by 14
  9. 2023Guiding Text-to-Text Privatization by Syntaxcited by 14
  10. 2021No Intruder, no Validity: Evaluation Criteria for Privacy-Preserving Text Anonymizationcited by 13
  11. 2024A Comparative Analysis of Word-Level Metric Differential Privacy: Benchmarking The Privacy-Utility Trade-offcited by 13
  12. 2024Evaluating Differentially Private Synthetic Data Generation in High-Stakes Domainscited by 12
  13. 2022Textwash -- automated open-source text anonymisationcited by 11
  14. 2024Thinking Outside of the Differential Privacy Box: A Case Study in Text Privatization with Language Model Promptingcited by 11
  15. 20241-Diffractor: Efficient and Utility-Preserving Text Obfuscation Leveraging Word-Level Metric Differential Privacycited by 10
  16. 2024On the Vulnerability of Text Sanitizationcited by 9
  17. 2024NAP^2: A Benchmark for Naturalness and Privacy-Preserving Text Rewriting by Learning from Humancited by 8
  18. 2022Stop the Spread: A Contextual Integrity Perspective on the Appropriateness of COVID-19 Vaccination Certificatescited by 4
  19. 2025Synthetic Data Can Mislead Evaluations: Membership Inference as Machine Text Detectioncited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 5.

5 papers.

  1. 2026A Survey on Evaluating Quality and Trustworthiness in LLM-Generated Datacited by 6
  2. 2026Synthetic Data for Veterinary EHR De-identification: Benefits, Limits, and Safety Trade-offs Under Fixed Computecited by 0
  3. 2026Distilling Human-Aligned Privacy Sensitivity Assessment from Large Language Modelscited by 0
  4. 2026Introducing the Privacy-HSD Trade-off: Hate Speech Detection, but not at the Cost of Privacycited by 0
  5. 2026PrivBench: A Holistic and Modular Benchmarking Platform for Evaluating Text-to-Text Privatizationcited by 0