Contextualized Privacy Defense for LLM Agents
On the map: LLM agents and tool use → Agent threats, guardrails and governance
Abstract (arXiv)
LLM agents increasingly act on users' personal information, yet existing privacy defenses remain limited in both design and adaptability. Most prior approaches rely on static or passive defenses, such as prompting and guarding. These paradigms are insufficient for supporting contextual, proactive privacy decisions in multi-step agent execution. We propose Contextualized Defense Instructing (CDI), a new privacy defense paradigm in which an instructor model generates step-specific, context-aware privacy guidance during execution, proactively shaping actions rather than merely constraining or vetoing them. Crucially, CDI is paired with an experience-driven optimization framework that trains the instructor via reinforcement learning (RL), where we convert failure trajectories with privacy violations into learning environments. We formalize baseline defenses and CDI as distinct intervention points in a canonical agent loop, and compare their privacy-helpfulness trade-offs within a unified simulation framework. Results show that our CDI consistently achieves a better balance between privacy preservation (94.2%) and helpfulness (80.6%) than baselines, with superior robustness to adversarial conditions and generalization.
Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use. Builds on 26 of its 37 references that are in the map. First posted on arXiv 3 Mar 2026.
Builds on
26 of 37 references.
Closest ancestors · 6
Same direction, newest first
- 2025Searching for Privacy Risks in LLM Agents via Simulation14 Aug
- 20251-2-3 Check: Enhancing Contextual Privacy in LLM via Multi-Agent Reasoning11 Aug
- 2025Progent: Securing AI Agents with Privilege Control16 Apr
- 2025Firewalls to Secure Dynamic LLM Agentic Networks3 Feb
- 2024When LLMs Go Online: The Emerging Threat of Web-Enabled LLMs18 Oct
- 2023Identifying the Risks of LM Agents with an LM-Emulated Sandbox25 Sep
Other references · 20
- 2020Language Models are Few-Shot Learnerscited by 32,419
- 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
- 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
- 2020Extracting Training Data from Large Language Modelscited by 2,189
- 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 2,006
- 2023CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Societycited by 1,395
- 2023Automatic Prompt Optimization with "Gradient Descent" and Beam Searchcited by 653
- 2024How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMscited by 573
- 2023DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Modelscited by 518
- 2024The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructionscited by 493
- 2023Can LLMs Keep a Secret? Testing Privacy Implications of Language Models via Contextual Integrity Theorycited by 230
- 2024SecAlign: Defending Against Prompt Injection with Preference Optimizationcited by 210
8 more
- 2024PrivacyLens: Evaluating Privacy Norm Awareness of Language Models in Actioncited by 171
- 2022Instruction Induction: From Few Examples to Natural Language Task Descriptionscited by 152
- 2024CoGenesis: A Framework Collaborating Large and Small Language Models for Secure Context-Aware Instruction Followingcited by 33
- 2025CIMemories: A Compositional Benchmark for Contextual Integrity of Persistent Memory in LLMscited by 26
- 2025Eliciting Language Model Behaviors with Investigator Agentscited by 23
- 2024Defending Against Social Engineering Attacks in the Age of LLMscited by 15
- 2025PPMI: Privacy-Preserving LLM Interaction with Socratic Chain-of-Thought Reasoning and Homomorphically Encrypted Vector Databasescited by 6
- 2025Zero-Shot Privacy-Aware Text Rewriting via Iterative Tree Searchcited by 3
Built on by
Citing papers per year: 2026: 7.
7 papers.
- 2026A Survey on Long-Term Memory Security in LLM Agents: Attacks, Defenses, and Governance Across the Memory Lifecyclecited by 27
- 2026Always-OnAgents:A Survey of Persistent Memory, State, and Governance in LLMAgentscited by 5
- 2026TRAP: Benchmark for Task-completion and Resistance to Active Privacy-extractioncited by 1
- 2026Selection-Invariant Communication Compilers for Privacy-Aware Multi-Agent LLM Workflowscited by 1
- 2026Confuse the Model, Control the Flow: Understanding and Mitigating Privacy Leakage from LLM Agents with Information Flow Controlcited by 0
- 2026Policy-Backed Selective Regeneration under Tainted Inter-Agent Communicationcited by 0
- 2026PrivacySkills: How Privacy Guidance Shapes Source Selection in LLM Agentscited by 0
- 2026PrivacySkills: How Privacy Guidance Shapes Source Selection in LLM Agents28 Sep 2026
- 2026Confuse the Model, Control the Flow: Understanding and Mitigating Privacy Leakage from LLM Agents with Information Flow Control12 Sep 2026
- 2026Selection-Invariant Communication Compilers for Privacy-Aware Multi-Agent LLM Workflows4 Aug 2026
- 2026Policy-Backed Selective Regeneration under Tainted Inter-Agent Communication2 Aug 2026
- 2026Always-OnAgents:A Survey of Persistent Memory, State, and Governance in LLMAgents29 Jun 2026
- 2026TRAP: Benchmark for Task-completion and Resistance to Active Privacy-extraction17 Jun 2026
- 2026A Survey on Long-Term Memory Security in LLM Agents: Attacks, Defenses, and Governance Across the Memory Lifecycle17 Apr 2026
Similar, not cited
Close in wording, no citation either way
- 2025Privacy-R1: Privacy-Aware Multi-LLM Agent Collaboration via Reinforcement Learning
- 2025Contextual Integrity in LLMs via Reasoning and Reinforcement Learning
- 2026PrivAct: Internalizing Contextual Privacy Preservation via Multi-Agent Preference Training
- 2025MAGPIE: A benchmark for Multi-AGent contextual PrIvacy Evaluation
- 2026It Takes Two: Complementary Self-Distillation for Contextual Integrity in LLMs
- 2025Position Paper: Rethinking Privacy in RL for Sequential Decision-making in the Age of LLMs
- 2025Beyond Jailbreaking: Auditing Contextual Privacy in LLM Agents
- 2026PrivacyAlign: Contextual Privacy Alignment for LLM Agents