# DP-MLM: Differentially Private Text Rewriting Using Masked Language Models

arXiv: https://arxiv.org/abs/2407.00637 · HTML: https://mlsanity.com/paper/2407.00637/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 32 references that are in the map.
- First posted on arXiv 30 Jun 2024.

## Builds on (most cited first, 20 of 20)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [Deep Learning with Differential Privacy](https://mlsanity.com/paper/1607.00133/) (2016)
- [Extracting Training Data from Large Language Models](https://mlsanity.com/paper/2012.07805/) (2020)
- [A large annotated corpus for learning natural language inference](https://mlsanity.com/paper/1508.05326/) (2015)
- [DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing](https://mlsanity.com/paper/2111.09543/) (2021)
- [What Does it Mean for a Language Model to Preserve Privacy?](https://mlsanity.com/paper/2202.05520/) (2022)
- [Privacy- and Utility-Preserving Textual Analysis via Calibrated Multivariate Perturbations](https://mlsanity.com/paper/1910.08902/) (2019)
- [Differential Privacy for Text Analytics via Natural Text Sanitization](https://mlsanity.com/paper/2106.01221/) (2021)
- [The Limits of Word Level Differential Privacy](https://mlsanity.com/paper/2205.02130/) (2022)
- [Locally Differentially Private Document Generation Using Zero Shot Prompting](https://mlsanity.com/paper/2310.16111/) (2023)
- [Generalised Differential Privacy for Text Document Processing](https://mlsanity.com/paper/1811.10256/) (2018)
- [A Customized Text Sanitization Mechanism with Differential Privacy](https://mlsanity.com/paper/2207.01193/) (2022)
- [Differentially Private Language Models Benefit from Public Pre-training](https://mlsanity.com/paper/2009.05886/) (2020)
- [TEM: High Utility Metric Differential Privacy on Text](https://mlsanity.com/paper/2107.07928/) (2021)
- [Learning Differentially Private Recurrent Language Models](https://mlsanity.com/paper/1710.06963/) (2017)
- [A Comparative Analysis of Word-Level Metric Differential Privacy: Benchmarking The Privacy-Utility Trade-off](https://mlsanity.com/paper/2404.03324/) (2024)
- [1-Diffractor: Efficient and Utility-Preserving Text Obfuscation Leveraging Word-Level Metric Differential Privacy](https://mlsanity.com/paper/2405.01678/) (2024)

## Built on by (most cited first, 24 of 24)

- [On the Impact of Noise in Differentially Private Text Rewriting](https://mlsanity.com/paper/2501.19022/) (2025)
- [Estimating Privacy Leakage of Augmented Contextual Knowledge in Language Models](https://mlsanity.com/paper/2410.03026/) (2024)
- [A Survey on Current Trends and Recent Advances in Text Anonymization](https://mlsanity.com/paper/2508.21587/) (2025)
- [How do we measure privacy in text? A survey of text anonymization metrics](https://mlsanity.com/paper/2512.01109/) (2025)
- [Leveraging Semantic Triples for Private Document Generation with Local Differential Privacy Guarantees](https://mlsanity.com/paper/2508.20736/) (2025)
- [Zero-Shot Privacy-Aware Text Rewriting via Iterative Tree Search](https://mlsanity.com/paper/2509.20838/) (2025)
- [DP-GTR: Differentially Private Prompt Protection via Group Text Rewriting](https://mlsanity.com/paper/2503.04990/) (2025)
- [Inspecting the Representation Manifold of Differentially-Private Text](https://mlsanity.com/paper/2503.14991/) (2025)
- [DP2Unlearning: An Efficient and Guaranteed Unlearning Framework for LLMs](https://mlsanity.com/paper/2504.13774/) (2025)
- [Investigating User Perspectives on Differentially Private Text Privatization](https://mlsanity.com/paper/2503.09338/) (2025)
- [Forget What's Sensitive, Remember What Matters: Token-Level Differential Privacy in Memory Sculpting for Continual Learning](https://mlsanity.com/paper/2509.12958/) (2025)
- [Adaptive Text Anonymization: Learning Privacy-Utility Trade-offs via Prompt Optimization](https://mlsanity.com/paper/2602.20743/) (2026)
- [Lexical Substitution is not Synonym Substitution: On the Importance of Producing Contextually Relevant Word Substitutes](https://mlsanity.com/paper/2502.04173/) (2025)
- [When Explainability Meets Privacy: An Investigation at the Intersection of Post-hoc Explainability and Differential Privacy in the Context of Natural Language Processing](https://mlsanity.com/paper/2508.10482/) (2025)
- [RL-Finetuned LLMs for Privacy-Preserving Synthetic Rewriting](https://mlsanity.com/paper/2508.19286/) (2025)
- [Protecting De-identified Documents from Search-based Linkage Attacks](https://mlsanity.com/paper/2510.06383/) (2025)
- [Beyond Theoretical Bounds: Empirical Privacy Loss Calibration for Text Rewriting Under Local Differential Privacy](https://mlsanity.com/paper/2603.22968/) (2026)
- [A Case Study on the Impact of Anonymization Along the RAG Pipeline](https://mlsanity.com/paper/2604.15958/) (2026)
- [LLM Anonymization Against Agentic Re-Identification](https://mlsanity.com/paper/2605.30848/) (2026)
- [Introducing the Privacy-HSD Trade-off: Hate Speech Detection, but not at the Cost of Privacy](https://mlsanity.com/paper/2608.19006/) (2026)
- [PrivBench: A Holistic and Modular Benchmarking Platform for Evaluating Text-to-Text Privatization](https://mlsanity.com/paper/2608.29624/) (2026)
- [I Am No One: Style-Aware Paraphrasing for Text Anonymization](https://mlsanity.com/paper/2609.12341/) (2026)
- [DP-IPI: A Hybrid Differential Privacy Text Rewriting Mechanism for Indirect Personal Identifiers in Clinical Texts](https://mlsanity.com/paper/2609.29684/) (2026)
- [Decoupling Knowledge and Privacy: Post-Task Self-Distillation Replay for LLM Continual Learning](https://mlsanity.com/paper/2609.29711/) (2026)

## Abstract (arXiv)

The task of text privatization using Differential Privacy has recently taken the form of textittext rewriting, in which an input text is obfuscated via the use of generative (large) language models. While these methods have shown promising results in the ability to preserve privacy, these methods rely on autoregressive models which lack a mechanism to contextualize the private rewriting process. In response to this, we propose DP-MLM, a new method for differentially private text rewriting based on leveraging masked language models (MLMs) to rewrite text in a semantically similar textitand obfuscated manner. We accomplish this with a simple contextualization technique, whereby we rewrite a text one token at a time. We find that utilizing encoder-only MLMs provides better utility preservation at lower varepsilon levels, as compared to previous methods relying on larger models with a decoder. In addition, MLMs allow for greater customization of the rewriting mechanism, as opposed to generative approaches. We make the code for DP-MLM public and reusable, found at https://github.com/sjmeis/DPMLM .