# Thinking Outside of the Differential Privacy Box: A Case Study in Text Privatization with Language Model Prompting

arXiv: https://arxiv.org/abs/2410.00751 · HTML: https://mlsanity.com/paper/2410.00751/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning.
- Builds on 4 of its 36 references that are in the map.
- First posted on arXiv 1 Oct 2024.

## Placed in

- LLM reasoning / [Reasoning for agents, planning and code](https://mlsanity.com/lr/reasoning-for-agents-planning-and-code/)

## Builds on (most cited first, 4 of 4)

- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Scaling Instruction-Finetuned Language Models](https://mlsanity.com/paper/2210.11416/) (2022)
- [DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing](https://mlsanity.com/paper/2111.09543/) (2021)
- [Towards General Text Embeddings with Multi-stage Contrastive Learning](https://mlsanity.com/paper/2308.03281/) (2023)

## Built on by (most cited first, 11 of 11)

- [LLM-as-a-Judge for Privacy Evaluation? Exploring the Alignment of Human and LLM Perceptions of Privacy in Textual Data](https://mlsanity.com/paper/2508.12158/) (2025)
- [On the Impact of Noise in Differentially Private Text Rewriting](https://mlsanity.com/paper/2501.19022/) (2025)
- [Spend Your Budget Wisely: Towards an Intelligent Distribution of the Privacy Budget in Differentially Private Text Rewriting](https://mlsanity.com/paper/2503.22379/) (2025)
- [How do we measure privacy in text? A survey of text anonymization metrics](https://mlsanity.com/paper/2512.01109/) (2025)
- [Leveraging Semantic Triples for Private Document Generation with Local Differential Privacy Guarantees](https://mlsanity.com/paper/2508.20736/) (2025)
- [STAMP: Selective Task-Aware Mechanism for Text Privacy](https://mlsanity.com/paper/2603.12237/) (2026)
- [Investigating User Perspectives on Differentially Private Text Privatization](https://mlsanity.com/paper/2503.09338/) (2025)
- [With Privacy, Size Matters: On the Importance of Dataset Size in Differentially Private Text Rewriting](https://mlsanity.com/paper/2511.00487/) (2025)
- [The Conundrum of Trustworthy Research on Attacking Personally Identifiable Information Removal Techniques](https://mlsanity.com/paper/2603.08207/) (2026)
- [PrivBench: A Holistic and Modular Benchmarking Platform for Evaluating Text-to-Text Privatization](https://mlsanity.com/paper/2608.29624/) (2026)
- [I Am No One: Style-Aware Paraphrasing for Text Anonymization](https://mlsanity.com/paper/2609.12341/) (2026)

## Abstract (arXiv)

The field of privacy-preserving Natural Language Processing has risen in popularity, particularly at a time when concerns about privacy grow with the proliferation of Large Language Models. One solution consistently appearing in recent literature has been the integration of Differential Privacy (DP) into NLP techniques. In this paper, we take these approaches into critical view, discussing the restrictions that DP integration imposes, as well as bring to light the challenges that such restrictions entail. To accomplish this, we focus on DP-Prompt, a recent method for text privatization leveraging language models to rewrite texts. In particular, we explore this rewriting task in multiple scenarios, both with DP and without DP. To drive the discussion on the merits of DP in NLP, we conduct empirical utility and privacy experiments. Our results demonstrate the need for more discussion on the usability of DP in NLP and its benefits over non-DP approaches.