# Anti-adversarial Learning: Desensitizing Prompts for Large Language Models

arXiv: https://arxiv.org/abs/2505.01273 · HTML: https://mlsanity.com/paper/2505.01273/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 28 references that are in the map.
- First posted on arXiv 25 Apr 2025.

## Builds on (most cited first, 18 of 18)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [Character-level Convolutional Networks for Text Classification](https://mlsanity.com/paper/1509.01626/) (2015)
- [Large Language Models Are Human-Level Prompt Engineers](https://mlsanity.com/paper/2211.01910/) (2022)
- [The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networks](https://mlsanity.com/paper/1802.08232/) (2018)
- [Automatic Prompt Optimization with "Gradient Descent" and Beam Search](https://mlsanity.com/paper/2305.03495/) (2023)
- [Generating Natural Language Adversarial Examples](https://mlsanity.com/paper/1804.07998/) (2018)
- [Can LLMs Keep a Secret? Testing Privacy Implications of Language Models via Contextual Integrity Theory](https://mlsanity.com/paper/2310.17884/) (2023)
- [Differential Privacy for Text Analytics via Natural Text Sanitization](https://mlsanity.com/paper/2106.01221/) (2021)
- [Locally Differentially Private Document Generation Using Zero Shot Prompting](https://mlsanity.com/paper/2310.16111/) (2023)
- [DP-OPT: Make Large Language Model Your Privacy-Preserving Prompt Engineer](https://mlsanity.com/paper/2312.03724/) (2023)
- [Natural Language Understanding with Privacy-Preserving BERT](https://mlsanity.com/paper/2104.07504/) (2021)
- [Hide and Seek (HaS): A Lightweight Framework for Prompt Privacy Protection](https://mlsanity.com/paper/2309.03057/) (2023)
- [D4: a Chinese Dialogue Dataset for Depression-Diagnosis-Oriented Chat](https://mlsanity.com/paper/2205.11764/) (2022)
- [CAPE: Context-Aware Private Embeddings for Private Language Learning](https://mlsanity.com/paper/2108.12318/) (2021)
- [No Free Lunch Theorem for Privacy-Preserving LLM Inference](https://mlsanity.com/paper/2405.20681/) (2024)
- [Data-adaptive Differentially Private Prompt Synthesis for In-Context Learning](https://mlsanity.com/paper/2410.12085/) (2024)
- [DePrompt: Desensitization and Evaluation of Personal Identifiable Information in Large Language Model Prompts](https://mlsanity.com/paper/2408.08930/) (2024)

## Built on by (most cited first, 2 of 2)

- [PrivScope: Task-scoped Disclosure Control for Hybrid Agentic Systems](https://mlsanity.com/paper/2605.16630/) (2026)
- [Combating Instruction Conflict via Energy-Driven Latent Conflict Detection](https://mlsanity.com/paper/2609.08646/) (2026)

## Abstract (arXiv)

With the widespread use of LLMs, preserving privacy in user prompts has become crucial, as prompts risk exposing privacy and sensitive data to the cloud LLMs. Traditional techniques like homomorphic encryption, secure multi-party computation, and federated learning face challenges due to heavy computational costs and user participation requirements, limiting their applicability in LLM scenarios. In this paper, we propose PromptObfus, a novel method for desensitizing LLM prompts. The core idea of PromptObfus is "anti-adversarial" learning, which perturbs privacy words in the prompt to obscure sensitive information while retaining the stability of model predictions. Specifically, PromptObfus frames prompt desensitization as a masked language modeling task, replacing privacy-sensitive terms with a [MASK] token. A desensitization model is trained to generate candidate replacements for each masked position. These candidates are subsequently selected based on gradient feedback from a surrogate model, ensuring minimal disruption to the task output. We demonstrate the effectiveness of our approach on three NLP tasks. Results show that PromptObfus effectively prevents privacy inference from remote LLMs while preserving task performance.