# AgentPrivArena: Evaluating and Auditing Real-world AI Agent Privacy

arXiv: https://arxiv.org/abs/2610.06454 · HTML: https://mlsanity.com/paper/2610.06454/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use.
- Builds on 9 of its 31 references that are in the map.
- First posted on arXiv 5 Oct 2026.

## Placed in

- LLM agents and tool use / [Agent threats, guardrails and governance](https://mlsanity.com/agents/agent-threats-guardrails-and-governance/)

## Builds on (most cited first, 9 of 9)

- [TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Tasks](https://mlsanity.com/paper/2412.14161/) (2024)
- [Defeating Prompt Injections by Design](https://mlsanity.com/paper/2503.18813/) (2025)
- [Progent: Securing AI Agents with Privilege Control](https://mlsanity.com/paper/2504.11703/) (2025)
- [AirGapAgent: Protecting Privacy-Conscious Conversational Agents](https://mlsanity.com/paper/2405.05175/) (2024)
- [Contextual Integrity in LLMs via Reasoning and Reinforcement Learning](https://mlsanity.com/paper/2506.04245/) (2025)
- [CI-Work: Benchmarking Contextual Integrity in Enterprise LLM Agents](https://mlsanity.com/paper/2604.21308/) (2026)
- [PrivacyPeek: Auditing What LLM-Based Agents Acquire, Not Just What They Say](https://mlsanity.com/paper/2606.00152/) (2026)
- [MuPPET: A Benchmark for Contextual Privacy of LLM Assistants in Multi-Party Conversations](https://mlsanity.com/paper/2606.23217/) (2026)
- [PiSAs: Benchmarking Contextual Integrity in Multi-User Agentic Systems](https://mlsanity.com/paper/2607.05318/) (2026)

## Abstract (arXiv)

The rapid advancement of LLM agents has enabled systems to autonomously perform complex tasks through external tools, but their growing access to personal data introduces significant privacy risks. Existing benchmarks primarily evaluate LLM agent privacy through simulated trajectories and outcome-based metrics, limiting their ability to capture privacy risks arising during multi-step agent execution. In this work, we introduce AgentPrivArena, a framework for evaluating privacy risks in realistic LLM agent workflows. AgentPrivArena integrates authentic MCP tools and self-hosted services within a reproducible execution environment. We further propose trajectory-level privacy metrics that quantify unnecessary information access beyond final response leakage. Building on this framework, we introduce AgentPrivAudit, a runtime auditing approach for monitoring privacy violations during agent execution. Extensive experiments on state-of-the-art LLM agents reveal substantial privacy risks overlooked by existing evaluation paradigms, highlighting the importance of trajectory-level auditing for trustworthy agent deployment.