# PrivBench: A Holistic and Modular Benchmarking Platform for Evaluating Text-to-Text Privatization

arXiv: https://arxiv.org/abs/2608.29624 · HTML: https://mlsanity.com/paper/2608.29624/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 105 of its 205 references that are in the map.
- First posted on arXiv 30 Aug 2026.

## Builds on (most cited first, 30 of 105)

- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [Character-level Convolutional Networks for Text Classification](https://mlsanity.com/paper/1509.01626/) (2015)
- [PubMedQA: A Dataset for Biomedical Research Question Answering](https://mlsanity.com/paper/1909.06146/) (2019)
- [Differentially Private Fine-tuning of Language Models](https://mlsanity.com/paper/2110.06500/) (2021)
- [PrivacyLens: Evaluating Privacy Norm Awareness of Language Models in Action](https://mlsanity.com/paper/2409.00138/) (2024)
- [OpenFedLLM: Training Large Language Models on Decentralized Private Data via Federated Learning](https://mlsanity.com/paper/2402.06954/) (2024)
- [Style Transfer Through Back-Translation](https://mlsanity.com/paper/1804.09000/) (2018)
- [Privacy- and Utility-Preserving Textual Analysis via Calibrated Multivariate Perturbations](https://mlsanity.com/paper/1910.08902/) (2019)
- [Synthetic Text Generation with Differential Privacy: A Simple and Practical Recipe](https://mlsanity.com/paper/2210.14348/) (2022)
- [Privacy-Preserving In-Context Learning with Differentially Private Few-Shot Generation](https://mlsanity.com/paper/2309.11765/) (2023)
- [Differential Privacy for Text Analytics via Natural Text Sanitization](https://mlsanity.com/paper/2106.01221/) (2021)
- [Differentially Private Synthetic Data via Foundation Model APIs 2: Text](https://mlsanity.com/paper/2403.01749/) (2024)
- [Towards Robust and Privacy-preserving Text Representations](https://mlsanity.com/paper/1805.06093/) (2018)
- [Differentially Private Language Models for Secure Data Sharing](https://mlsanity.com/paper/2210.13918/) (2022)
- [The Limits of Word Level Differential Privacy](https://mlsanity.com/paper/2205.02130/) (2022)
- [Locally Differentially Private Document Generation Using Zero Shot Prompting](https://mlsanity.com/paper/2310.16111/) (2023)
- [Selective Differential Privacy for Language Modeling](https://mlsanity.com/paper/2108.12944/) (2021)
- [THE-X: Privacy-Preserving Transformer Inference with Homomorphic Encryption](https://mlsanity.com/paper/2206.00216/) (2022)
- [Natural Language Understanding with Privacy-Preserving BERT](https://mlsanity.com/paper/2104.07504/) (2021)
- [Generalised Differential Privacy for Text Document Processing](https://mlsanity.com/paper/1811.10256/) (2018)
- [A Customized Text Sanitization Mechanism with Differential Privacy](https://mlsanity.com/paper/2207.01193/) (2022)
- [Leveraging Hierarchical Representations for Preserving Privacy and Utility in Text](https://mlsanity.com/paper/1910.08917/) (2019)
- [Differentially Private Language Models Benefit from Public Pre-training](https://mlsanity.com/paper/2009.05886/) (2020)
- [TEM: High Utility Metric Differential Privacy on Text](https://mlsanity.com/paper/2107.07928/) (2021)
- [A Differentially Private Text Perturbation Method Using a Regularized Mahalanobis Metric](https://mlsanity.com/paper/2010.11947/) (2020)
- [Attribution and Obfuscation of Neural Text Authorship: A Data Mining Perspective](https://mlsanity.com/paper/2210.10488/) (2022)
- [PrivLM-Bench: A Multi-level Privacy Evaluation Benchmark for Language Models](https://mlsanity.com/paper/2311.04044/) (2023)
- [Privacy Adhering Machine Un-learning in NLP](https://mlsanity.com/paper/2212.09573/) (2022)

## Abstract (arXiv)

Natural Language Processing methods have enabled novel solutions and advances in the field of privacy, particularly in the sub-domain of text-to-text privatization, where the goal is to transform a sensitive input text into a privatized output by ideally masking (in)directly identifiable or otherwise private information. The evaluation of text-to-text privatization, however, is not straightforward, and the extant literature has utilized a myriad of techniques and metrics to quantify the privacy-preserving capabilities of privatization methods. Seeking to unify the evaluation of text-to-text privatization, we introduce PrivBench, a holistic and modular benchmarking platform for researchers and practitioners working on text privatization. PrivBench is holistic in that it evaluates privatization on a series of defined desiderata, which are structured into modules. PrivBench is not only modular but also extensible, allowing for future updates and benchmark versions. PrivBench is user-centered and promotes competition via real-time evaluation and a live public leaderboard. The platform is free to use and openly accessible at https://privbench.com/.