# The Text Anonymization Benchmark (TAB): A Dedicated Corpus and Evaluation Framework for Text Anonymization

arXiv: https://arxiv.org/abs/2202.00443 · HTML: https://mlsanity.com/paper/2202.00443/

## Key facts

- Cited by 95 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 116 references that are in the map.
- First posted on arXiv 25 Jan 2022.

## Builds on (most cited first, 26 of 26)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [Membership Inference Attacks against Machine Learning Models](https://mlsanity.com/paper/1610.05820/) (2016)
- [BioBERT: a pre-trained biomedical language representation model for biomedical text mining](https://mlsanity.com/paper/1901.08746/) (2019)
- [SciBERT: A Pretrained Language Model for Scientific Text](https://mlsanity.com/paper/1903.10676/) (2019)
- [Neural Architectures for Named Entity Recognition](https://mlsanity.com/paper/1603.01360/) (2016)
- [Large Language Models Can Be Strong Differentially Private Learners](https://mlsanity.com/paper/2110.05679/) (2021)
- [End-to-end Neural Coreference Resolution](https://mlsanity.com/paper/1707.07045/) (2017)
- [Named Entity Recognition with Bidirectional LSTM-CNNs](https://mlsanity.com/paper/1511.08308/) (2015)
- [Demographic Dialectal Variation in Social Media: A Case Study of African-American English](https://mlsanity.com/paper/1608.08868/) (2016)
- [Adversarial Removal of Demographic Attributes from Text Data](https://mlsanity.com/paper/1808.06640/) (2018)
- [De-identification of Patient Notes with Recurrent Neural Networks](https://mlsanity.com/paper/1606.03475/) (2016)
- [Towards Robust and Privacy-preserving Text Representations](https://mlsanity.com/paper/1805.06093/) (2018)
- [Generalised Differential Privacy for Text Document Processing](https://mlsanity.com/paper/1811.10256/) (2018)
- [Leveraging Hierarchical Representations for Preserving Privacy and Utility in Text](https://mlsanity.com/paper/1910.08917/) (2019)
- [TextHide: Tackling Data Privacy in Language Understanding Tasks](https://mlsanity.com/paper/2010.06053/) (2020)
- [Anaphora and Coreference Resolution: A Review](https://mlsanity.com/paper/1805.11824/) (2018)
- [ADePT: Auto-encoder based Differentially Private Text Transformation](https://mlsanity.com/paper/2102.01502/) (2021)
- [C-sanitized: a privacy model for document redaction and sanitization](https://mlsanity.com/paper/1406.4285/) (2014)
- [When differential privacy meets NLP: The devil is in the detail](https://mlsanity.com/paper/2109.03175/) (2021)
- [No Intruder, no Validity: Evaluation Criteria for Privacy-Preserving Text Anonymization](https://mlsanity.com/paper/2103.09263/) (2021)
- [A Privacy-Preserving Approach to Extraction of Personal Information through Automatic Annotation and Federated Learning](https://mlsanity.com/paper/2105.09198/) (2021)
- [De-identification of Privacy-related Entities in Job Postings](https://mlsanity.com/paper/2105.11223/) (2021)
- [Toward sensitive document release with privacy guarantees](https://mlsanity.com/paper/1701.00436/) (2017)
- [A survey of automatic de-identification of longitudinal clinical narratives](https://mlsanity.com/paper/1810.06765/) (2018)

## Built on by (most cited first, 30 of 95)

- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023)
- [Analyzing Leakage of Personally Identifiable Information in Language Models](https://mlsanity.com/paper/2302.00539/) (2023)
- [Clio: Privacy-Preserving Insights into Real-World AI Use](https://mlsanity.com/paper/2412.13678/) (2024)
- [Reducing Privacy Risks in Online Self-Disclosures with Language Models](https://mlsanity.com/paper/2311.09538/) (2023)
- [LLM-PBE: Assessing Data Privacy in Large Language Models](https://mlsanity.com/paper/2408.12787/) (2024)
- [Privacy-Preserving Instructions for Aligning Large Language Models](https://mlsanity.com/paper/2402.13659/) (2024)
- [IncogniText: Privacy-enhancing Conditional Text Anonymization via LLM-based Private Attribute Randomization](https://mlsanity.com/paper/2407.02956/) (2024)
- [Large Language Models are Advanced Anonymizers](https://mlsanity.com/paper/2402.13846/) (2024)
- [Differentially Private Natural Language Models: Recent Advances and Future Directions](https://mlsanity.com/paper/2301.09112/) (2023)
- [Privacy- and Utility-Preserving NLP with Anonymized Data: A case study of Pseudonymization](https://mlsanity.com/paper/2306.05561/) (2023)
- [A False Sense of Privacy: Evaluating Textual Data Sanitization Beyond Surface-level Privacy Leakage](https://mlsanity.com/paper/2504.21035/) (2025)
- [Tokens for Learning, Tokens for Unlearning: Mitigating Membership Inference Attacks in Large Language Models via Dual-Purpose Training](https://mlsanity.com/paper/2502.19726/) (2025)
- [Evaluating LLM-based Personal Information Extraction and Countermeasures](https://mlsanity.com/paper/2408.07291/) (2024)
- [Unsupervised Text Deidentification](https://mlsanity.com/paper/2210.11528/) (2022)
- [PII-Bench: Evaluating Query-Aware Privacy Protection Systems](https://mlsanity.com/paper/2502.18545/) (2025)
- [Privacy-Preserving Models for Legal Natural Language Processing](https://mlsanity.com/paper/2211.02956/) (2022)
- [Does fine-tuning GPT-3 with the OpenAI API leak personally-identifiable information?](https://mlsanity.com/paper/2307.16382/) (2023)
- [Beyond Document Page Classification: Design, Datasets, and Challenges](https://mlsanity.com/paper/2308.12896/) (2023)
- [Text Sanitization Beyond Specific Domains: Zero-Shot Redaction & Substitution with Large Language Models](https://mlsanity.com/paper/2311.10785/) (2023)
- [Robust Utility-Preserving Text Anonymization Based on Large Language Models](https://mlsanity.com/paper/2407.11770/) (2024)
- [SOFT: Selective Data Obfuscation for Protecting LLM Fine-tuning against Membership Inference Attacks](https://mlsanity.com/paper/2506.10424/) (2025)
- [InvisibleInk: High-Utility and Low-Cost Text Generation with Differential Privacy](https://mlsanity.com/paper/2507.02974/) (2025)
- [LLM-as-a-Judge for Privacy Evaluation? Exploring the Alignment of Human and LLM Perceptions of Privacy in Textual Data](https://mlsanity.com/paper/2508.12158/) (2025)
- [Gradient-Free Privacy Leakage in Federated Language Models through Selective Weight Tampering](https://mlsanity.com/paper/2310.16152/) (2023)
- [Bootstrapping Text Anonymization Models with Distant Supervision](https://mlsanity.com/paper/2205.06895/) (2022)
- [NAP^2: A Benchmark for Naturalness and Privacy-Preserving Text Rewriting by Learning from Human](https://mlsanity.com/paper/2406.03749/) (2024)
- [Casper: Prompt Sanitization for Protecting User Privacy in Web-Based Large Language Models](https://mlsanity.com/paper/2408.07004/) (2024)
- [Addressing Resource and Privacy Constraints in Semantic Parsing Through Data Augmentation](https://mlsanity.com/paper/2205.08675/) (2022)
- [Unlocking the Potential of Large Language Models for Clinical Text Anonymization: A Comparative Study](https://mlsanity.com/paper/2406.00062/) (2024)
- [SynthTextEval: Synthetic Text Data Generation and Evaluation for High-Stakes Domains](https://mlsanity.com/paper/2507.07229/) (2025)

## Abstract (arXiv)

We present a novel benchmark and associated evaluation metrics for assessing the performance of text anonymization methods. Text anonymization, defined as the task of editing a text document to prevent the disclosure of personal information, currently suffers from a shortage of privacy-oriented annotated text resources, making it difficult to properly evaluate the level of privacy protection offered by various anonymization methods. This paper presents TAB (Text Anonymization Benchmark), a new, open-source annotated corpus developed to address this shortage. The corpus comprises 1,268 English-language court cases from the European Court of Human Rights (ECHR) enriched with comprehensive annotations about the personal information appearing in each document, including their semantic category, identifier type, confidential attributes, and co-reference relations. Compared to previous work, the TAB corpus is designed to go beyond traditional de-identification (which is limited to the detection of predefined semantic categories), and explicitly marks which text spans ought to be masked in order to conceal the identity of the person to be protected. Along with presenting the corpus and its annotation layers, we also propose a set of evaluation metrics that are specifically tailored towards measuring the performance of text anonymization, both in terms of privacy protection and utility preservation. We illustrate the use of the benchmark and the proposed metrics by assessing the empirical performance of several baseline text anonymization models. The full corpus along with its privacy-oriented annotation guidelines, evaluation scripts and baseline models are available on: https://github.com/NorskRegnesentral/text-anonymisation-benchmark