# PrivacyBench: A Conversational Benchmark for Evaluating Privacy in Personalized AI

arXiv: https://arxiv.org/abs/2512.24848 · HTML: https://mlsanity.com/paper/2512.24848/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 31 references that are in the map.
- First posted on arXiv 31 Dec 2025.

## Builds on (most cited first, 17 of 17)

- [Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities](https://mlsanity.com/paper/2507.06261/) (2025)
- [G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment](https://mlsanity.com/paper/2303.16634/) (2023)
- [Extracting Training Data from Large Language Models](https://mlsanity.com/paper/2012.07805/) (2020)
- [Jailbreaking ChatGPT via Prompt Engineering: An Empirical Study](https://mlsanity.com/paper/2305.13860/) (2023)
- [LaMP: When Large Language Models Meet Personalization](https://mlsanity.com/paper/2304.11406/) (2023)
- [Scaling Synthetic Data Creation with 1,000,000,000 Personas](https://mlsanity.com/paper/2406.20094/) (2024)
- [Multi-step Jailbreaking Privacy Attacks on ChatGPT](https://mlsanity.com/paper/2304.05197/) (2023)
- [Beyond Goldfish Memory: Long-Term Open-Domain Conversation](https://mlsanity.com/paper/2107.07567/) (2021)
- [Can LLMs Keep a Secret? Testing Privacy Implications of Language Models via Contextual Integrity Theory](https://mlsanity.com/paper/2310.17884/) (2023)
- [Phi-4-reasoning Technical Report](https://mlsanity.com/paper/2504.21318/) (2025)
- [Differential Privacy and Machine Learning: a Survey and Review](https://mlsanity.com/paper/1412.7584/) (2014)
- [Great Models Think Alike and this Undermines AI Oversight](https://mlsanity.com/paper/2502.04313/) (2025)
- [Personalized Generation In Large Model Era: A Survey](https://mlsanity.com/paper/2503.02614/) (2025)
- [LaMP-QA: A Benchmark for Personalized Long-form Question Answering](https://mlsanity.com/paper/2506.00137/) (2025)
- [Exploring Safety-Utility Trade-Offs in Personalized Language Models](https://mlsanity.com/paper/2406.11107/) (2024)
- [ExPerT: Effective and Explainable Evaluation of Personalized Long-Form Text Generation](https://mlsanity.com/paper/2501.14956/) (2025)
- [Privacy Checklist: Privacy Violation Detection Grounding on Contextual Integrity Theory](https://mlsanity.com/paper/2408.10053/) (2024)

## Built on by (most cited first, 12 of 12)

- [Toward Personalized LLM-Powered Agents: Foundations, Evaluation, and Future Directions](https://mlsanity.com/paper/2602.22680/) (2026)
- [MemPrivacy: Privacy-Preserving Personalized Memory Management for Edge-Cloud Agents](https://mlsanity.com/paper/2605.09530/) (2026)
- [Remembering More, Risking More: Longitudinal Safety Risks in Memory-Equipped LLM Agents](https://mlsanity.com/paper/2605.17830/) (2026)
- [When Agents Learn to Be You: Benchmarking Privacy Leakage, Impersonation Risk, and Defenses in Persona Skills](https://mlsanity.com/paper/2608.03700/) (2026)
- [PrivacyAlign: Contextual Privacy Alignment for LLM Agents](https://mlsanity.com/paper/2606.21710/) (2026)
- [LUNAR: Benchmarking Personalized Large Language Models on UNiversal User BehAvioR Logs](https://mlsanity.com/paper/2608.05246/) (2026)
- [It Takes Two: Complementary Self-Distillation for Contextual Integrity in LLMs](https://mlsanity.com/paper/2605.20258/) (2026)
- [TRAP: Benchmark for Task-completion and Resistance to Active Privacy-extraction](https://mlsanity.com/paper/2606.18996/) (2026)
- [How Far Are VLMs from Privacy Awareness in the Physical World? An Empirical Study](https://mlsanity.com/paper/2605.05340/) (2026)
- [MemArena: An Ego-Centric Benchmark for On-Device Agentic Personal Memory Assistants at Scale](https://mlsanity.com/paper/2608.02613/) (2026)
- [Privacy in Personalized AI Is a System Property, Not Just a Model Property](https://mlsanity.com/paper/2609.38289/) (2026)
- [Loose Lips: Understanding Privacy Leakage in Agentic Dialogue](https://mlsanity.com/paper/2610.14071/) (2026)

## Abstract (arXiv)

Personalized AI agents rely on access to a user's digital footprint, which often includes sensitive data from private emails, chats and purchase histories. Yet this access creates a fundamental societal and privacy risk: systems lacking social-context awareness can unintentionally expose user secrets, threatening digital well-being. We introduce PrivacyBench, a benchmark with socially grounded datasets containing embedded secrets and a multi-turn conversational evaluation to measure secret preservation. Testing Retrieval-Augmented Generation (RAG) assistants reveals that they leak secrets in up to 26.56% of interactions. A privacy-aware prompt lowers leakage to 5.12%, yet this measure offers only partial mitigation. The retrieval mechanism continues to access sensitive data indiscriminately, which shifts the entire burden of privacy preservation onto the generator. This creates a single point of failure, rendering current architectures unsafe for wide-scale deployment. Our findings underscore the urgent need for structural, privacy-by-design safeguards to ensure an ethical and inclusive web for everyone.