# AgentSCOPE: Evaluating Contextual Privacy Across Agentic Workflows

arXiv: https://arxiv.org/abs/2603.04902 · HTML: https://mlsanity.com/paper/2603.04902/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use.
- Builds on 4 of its 6 references that are in the map.
- First posted on arXiv 5 Mar 2026.

## Placed in

- LLM agents and tool use / [Agent threats, guardrails and governance](https://mlsanity.com/agents/agent-threats-guardrails-and-governance/)

## Builds on (most cited first, 4 of 4)

- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [PrivacyLens: Evaluating Privacy Norm Awareness of Language Models in Action](https://mlsanity.com/paper/2409.00138/) (2024)
- [Privacy in Action: Towards Realistic Privacy Mitigation and Evaluation for LLM-Powered Agents](https://mlsanity.com/paper/2509.17488/) (2025)
- [Protecting Users From Themselves: Safeguarding Contextual Privacy in Interactions with Conversational Agents](https://mlsanity.com/paper/2502.18509/) (2025)

## Built on by (most cited first, 10 of 10)

- [Can LLM-as-a-Judge Reliably Verify Rubrics in Agentic Scenarios?](https://mlsanity.com/paper/2606.29920/) (2026)
- [When Agents Do Not Stop: Uncovering Infinite Agentic Loops in LLM Agents](https://mlsanity.com/paper/2607.01641/) (2026)
- [PrivacyPeek: Auditing What LLM-Based Agents Acquire, Not Just What They Say](https://mlsanity.com/paper/2606.00152/) (2026)
- [The Authorization-Execution Gap Is a Major Safety and Security Problem in Open-World Agents](https://mlsanity.com/paper/2605.11003/) (2026)
- [Plant, Persist, Trigger: Sleeper Attack on Large Language Model Agents](https://mlsanity.com/paper/2605.28201/) (2026)
- [MuPPET: A Benchmark for Contextual Privacy of LLM Assistants in Multi-Party Conversations](https://mlsanity.com/paper/2606.23217/) (2026)
- [Security and Privacy in Agentic AI: Grand Challenges and Future Directions](https://mlsanity.com/paper/2607.06608/) (2026)
- [Toward Trustworthy Autonomous Science: A Two-Year Community Roadmap](https://mlsanity.com/paper/2607.12113/) (2026)
- [MemArena: An Ego-Centric Benchmark for On-Device Agentic Personal Memory Assistants at Scale](https://mlsanity.com/paper/2608.02613/) (2026)
- [Empowering Edge Agents: A Systematic Analysis of Edge-Cloud Collaboration in LLM Agent Execution](https://mlsanity.com/paper/2610.17816/) (2026)

## Abstract (arXiv)

Agentic systems are increasingly acting on users' behalf, accessing calendars, email, and personal files to complete everyday tasks. Privacy evaluation for these systems has focused on the input and output boundaries, but each task involves several intermediate information flows, from agent queries to tool responses, that are not currently evaluated. We argue that every boundary in an agentic pipeline is a site of potential privacy violation and must be assessed independently. To support this, we introduce the Privacy Flow Graph, a Contextual Integrity-grounded framework that decomposes agentic execution into a sequence of information flows, each annotated with the five CI parameters, and traces violations to their point of origin. We present AgentSCOPE, a benchmark of 62 multi-tool scenarios across eight regulatory domains with ground truth at every pipeline stage. Our evaluation across seven state-of-the-art LLMs show that privacy violations in the pipeline occur in over 80% of scenarios, even when final outputs appear clean (24%), with most violations arising at the tool-response stage where APIs return sensitive data indiscriminately. These results indicate that output-level evaluation alone substantially underestimates the privacy risk of agentic systems.