# Privacy in Action: Towards Realistic Privacy Mitigation and Evaluation for LLM-Powered Agents

arXiv: https://arxiv.org/abs/2509.17488 · HTML: https://mlsanity.com/paper/2509.17488/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use.
- Builds on 15 of its 31 references that are in the map.
- First posted on arXiv 22 Sep 2025.

## Placed in

- LLM agents and tool use / [Agent threats, guardrails and governance](https://mlsanity.com/agents/agent-threats-guardrails-and-governance/)

## Builds on (most cited first, 15 of 15)

- [Toolformer: Language Models Can Teach Themselves to Use Tools](https://mlsanity.com/paper/2302.04761/) (2023)
- [TravelPlanner: A Benchmark for Real-World Planning with Language Agents](https://mlsanity.com/paper/2402.01622/) (2024)
- [Can LLMs Keep a Secret? Testing Privacy Implications of Language Models via Contextual Integrity Theory](https://mlsanity.com/paper/2310.17884/) (2023)
- [PrivacyLens: Evaluating Privacy Norm Awareness of Language Models in Action](https://mlsanity.com/paper/2409.00138/) (2024)
- [Political Compass or Spinning Arrow? Towards More Meaningful Evaluations for Values and Opinions in Large Language Models](https://mlsanity.com/paper/2402.16786/) (2024)
- [Adaptive Attacks Break Defenses Against Indirect Prompt Injection Attacks on LLM Agents](https://mlsanity.com/paper/2503.00061/) (2025)
- [MMedAgent: Learning to Use Medical Tools with Multi-modal Agent](https://mlsanity.com/paper/2407.02483/) (2024)
- [CultureBank: An Online Community-Driven Knowledge Base Towards Culturally Aware Language Technologies](https://mlsanity.com/paper/2404.15238/) (2024)
- [Federated Learning of Gboard Language Models with Differential Privacy](https://mlsanity.com/paper/2305.18465/) (2023)
- [Plan-Then-Execute: An Empirical Study of User Trust and Team Performance When Using LLM Agents As A Daily Assistant](https://mlsanity.com/paper/2502.01390/) (2025)
- [CI-Bench: Benchmarking Contextual Integrity of AI Assistants on Synthetic Data](https://mlsanity.com/paper/2409.13903/) (2024)
- [The Consensus Game: Language Model Generation via Equilibrium Search](https://mlsanity.com/paper/2310.09139/) (2023)
- [Safely Learning with Private Data: A Federated Learning Framework for Large Language Model](https://mlsanity.com/paper/2406.14898/) (2024)
- [Navigating the Unknown: A Chat-Based Collaborative Interface for Personalized Exploratory Tasks](https://mlsanity.com/paper/2410.24032/) (2024)
- [Privacy Evaluation Benchmarks for NLP Models](https://mlsanity.com/paper/2409.15868/) (2024)

## Built on by (most cited first, 30 of 36)

- [The Attack and Defense Landscape of Agentic AI: A Comprehensive Survey](https://mlsanity.com/paper/2603.11088/) (2026)
- [A Survey on Agentic Security: Applications, Threats and Defenses](https://mlsanity.com/paper/2510.06445/) (2025)
- [CI-Work: Benchmarking Contextual Integrity in Enterprise LLM Agents](https://mlsanity.com/paper/2604.21308/) (2026)
- [Agent Tools Orchestration Leaks More: Dataset, Benchmark, and Mitigation](https://mlsanity.com/paper/2512.16310/) (2025)
- [AgentSCOPE: Evaluating Contextual Privacy Across Agentic Workflows](https://mlsanity.com/paper/2603.04902/) (2026)
- [PrivAct: Internalizing Contextual Privacy Preservation via Multi-Agent Preference Training](https://mlsanity.com/paper/2602.13840/) (2026)
- [AgentRaft: Automated Detection of Data Over-Exposure in LLM Agents](https://mlsanity.com/paper/2603.07557/) (2026)
- [Long Context, Less Focus: A Scaling Gap in LLMs Revealed through Privacy and Personalization](https://mlsanity.com/paper/2602.15028/) (2026)
- [POLAR-Bench: A Diagnostic Benchmark for Privacy-Utility Trade-offs in LLM Agents](https://mlsanity.com/paper/2605.19127/) (2026)
- [MuPPET: A Benchmark for Contextual Privacy of LLM Assistants in Multi-Party Conversations](https://mlsanity.com/paper/2606.23217/) (2026)
- [MPCI-Bench: A Benchmark for Multimodal Pairwise Contextual Integrity Evaluation of Language Model Agents](https://mlsanity.com/paper/2601.08235/) (2026)
- [Minim: Privacy-Aware Minimal View for Agents via Trusted Local Sanitization](https://mlsanity.com/paper/2606.13949/) (2026)
- [PrivacyAlign: Contextual Privacy Alignment for LLM Agents](https://mlsanity.com/paper/2606.21710/) (2026)
- [User Perceptions vs. Proxy LLM Judges: Privacy and Helpfulness in LLM Responses to Privacy-Sensitive Scenarios](https://mlsanity.com/paper/2510.20721/) (2025)
- [AudAgent: Automated Auditing of Privacy Policy Compliance in AI Agents](https://mlsanity.com/paper/2511.07441/) (2025)
- [NeuroFilter: Activation-Based Guardrails for Privacy-Conscious LLM Agents](https://mlsanity.com/paper/2601.14660/) (2026)
- [Safer Reasoning Traces: Measuring and Mitigating Chain-of-Thought Leakage in LLMs](https://mlsanity.com/paper/2603.05618/) (2026)
- [It Takes Two: Complementary Self-Distillation for Contextual Integrity in LLMs](https://mlsanity.com/paper/2605.20258/) (2026)
- [MaskClaw: Edge-Side Personalized Privacy Arbitration for GUI Agents with Behavior-Driven Skill Evolution](https://mlsanity.com/paper/2605.28646/) (2026)
- [ToolMinimize: Auditing and Rewriting LLM Agent Tool Calls to Minimize Privacy Exposure](https://mlsanity.com/paper/2608.24957/) (2026)
- [PII-TRACE: A Benchmark for Context-Aware PII Detection in Multi-Turn LLM Conversations](https://mlsanity.com/paper/2609.22200/) (2026)
- [Say Something Else: Rethinking Contextual Privacy as Information Sufficiency](https://mlsanity.com/paper/2604.06409/) (2026)
- [Securing Computer-Use Agents: A Unified Architecture-Lifecycle Framework for Deployment-Grounded Reliability](https://mlsanity.com/paper/2605.07110/) (2026)
- [OCELOT: Inference-Leakage Budgets for Privacy-Preserving LLM Agents](https://mlsanity.com/paper/2606.12341/) (2026)
- [Agents That Know Too Much: A Data-Centric Survey of Privacy in LLM Agents](https://mlsanity.com/paper/2606.26627/) (2026)
- [MemArena: An Ego-Centric Benchmark for On-Device Agentic Personal Memory Assistants at Scale](https://mlsanity.com/paper/2608.02613/) (2026)
- [DP-MemView: A Memory Interface for Attribute-Level Transcript Privacy in Long-Term LLM Agents](https://mlsanity.com/paper/2608.03130/) (2026)
- [Rethinking Agent Security as a Networking Problem](https://mlsanity.com/paper/2608.12172/) (2026)
- [Inferring Hidden User Models from the Behavior of Personalized LLM Agents](https://mlsanity.com/paper/2609.03815/) (2026)
- [Confuse the Model, Control the Flow: Understanding and Mitigating Privacy Leakage from LLM Agents with Information Flow Control](https://mlsanity.com/paper/2609.14003/) (2026)

## Abstract (arXiv)

The increasing autonomy of LLM agents in handling sensitive communications, accelerated by Model Context Protocol (MCP) and Agent-to-Agent (A2A) frameworks, creates urgent privacy challenges. While recent work reveals significant gaps between LLMs' privacy Q&A performance and their agent behavior, existing benchmarks remain limited to static, simplified scenarios. We present PrivacyChecker, a model-agnostic, contextual integrity based mitigation approach that effectively reduces privacy leakage from 36.08% to 7.30% on DeepSeek-R1 and from 33.06% to 8.32% on GPT-4o, all while preserving task helpfulness. We also introduce PrivacyLens-Live, transforming static benchmarks into dynamic MCP and A2A environments that reveal substantially higher privacy risks in practical. Our modular mitigation approach integrates seamlessly into agent protocols through three deployment strategies, providing practical privacy protection for the emerging agentic ecosystem. Our data and code will be made available at https://aka.ms/privacy_in_action.