mlsanityData to 7 Oct 2026

AirGapAgent: Protecting Privacy-Conscious Conversational Agents

arXiv 2405.05175 ↗First posted 8 May 2024cs.CR cs.CL cs.LGcited by 90

On the map: LLM agents and tool use → Agent threats, guardrails and governance

Abstract (arXiv)

The growing use of large language model (LLM)-based conversational agents to manage sensitive user data raises significant privacy concerns. While these agents excel at understanding and acting on context, this capability can be exploited by malicious actors. We introduce a novel threat model where adversarial third-party apps manipulate the context of interaction to trick LLM-based agents into revealing private information not relevant to the task at hand. Grounded in the framework of contextual integrity, we introduce AirGapAgent, a privacy-conscious agent designed to prevent unintended data leakage by restricting the agent's access to only the data necessary for a specific task. Extensive experiments using Gemini, GPT, and Mistral models as agents validate our approach's effectiveness in mitigating this form of context hijacking while maintaining core agent functionality. For example, we show that a single-query context hijacking attack on a Gemini Ultra agent reduces its ability to protect user data from 94% to 45%, while an AirGapAgent achieves 97% protection, rendering the same attack ineffective.

Key facts. Cited by 90 papers in the map (data through 7 Oct 2026). Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use. Builds on 29 of its 63 references that are in the map. First posted on arXiv 8 May 2024.

Builds on

29 of 63 references.

References · 29

  1. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  2. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  3. 2023Mistral 7Bcited by 3,065
  4. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  5. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,203
  6. 2016Concrete Problems in AI Safetycited by 1,582
  7. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,565
  8. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,422
  9. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,326
  10. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,196
  11. 2022Red Teaming Language Models with Language Modelscited by 983
  12. 2019Universal Adversarial Triggers for Attacking and Analyzing NLPcited by 709
17 more
  1. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 702
  2. 2023DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Modelscited by 505
  3. 2024The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructionscited by 441
  4. 2023Scalable Extraction of Training Data from (Production) Language Modelscited by 415
  5. 2023Can LLMs Keep a Secret? Testing Privacy Implications of Language Models via Contextual Integrity Theorycited by 218
  6. 2022What Does it Mean for a Language Model to Preserve Privacy?cited by 194
  7. 2024Do Membership Inference Attacks Work on Large Language Models?cited by 188
  8. 2024Rethinking Interpretability in the Era of Large Language Modelscited by 110
  9. 2024The Ethics of Advanced AI Assistantscited by 103
  10. 2024A New Era in LLM Security: Exploring Security Concerns in Real-World LLM-based Systemscited by 74
  11. 2024Can LLMs Separate Instructions From Data? And What Do We Even Mean By That?cited by 70
  12. 2021TAG: Gradient Attack on Transformer-based Language Modelscited by 63
  13. 2024Teach LLMs to Phish: Stealing Private Information from Language Modelscited by 29
  14. 2019Context-Aware Local Differential Privacycited by 18
  15. 2023Limits for Learning with Language Modelscited by 17
  16. 2015Android Permissions Remystified: A Field Study on Contextual Integritycited by 15
  17. 2015Implicit Contextual Integrity in Online Social Networkscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 6; 2025: 38; 2026: 46.

90 papers.

Sort:

By area:

  1. 2025Defeating Prompt Injections by Designcited by 256
  2. 2024The Emerged Security and Privacy of LLM Agent: A Survey with Case Studiescited by 127
  3. 2025Progent: Securing AI Agents with Privilege Controlcited by 121
  4. 2026Skill-Inject: Measuring Agent Vulnerability to Skill File Attackscited by 82
  5. 2025AgentDAM: Privacy Leakage Evaluation for Autonomous Web Agentscited by 79
  6. 2025Design Patterns for Securing LLM Agents against Prompt Injectionscited by 67
  7. 2024Navigating the Risks: A Survey of Security, Privacy, and Ethics Threats in LLM-Based Agentscited by 63
  8. 2024Get my drift? Catching LLM Task Drift with Activation Deltascited by 61
  9. 2025Contextual Agent Security: A Policy for Every Purposecited by 55
  10. 2025The Obvious Invisible Threat: LLM-Powered GUI Agents' Vulnerability to Fine-Print Injectionscited by 55
  11. 2025ACE: A Security Architecture for LLM-Integrated App Systemscited by 52
  12. 2025SAGA: A Security Architecture for Governing AI Agentic Systemscited by 45
38 more
  1. 2025Fully Autonomous AI Agents Should Not be Developedcited by 43
  2. 2024Operationalizing Contextual Integrity in Privacy-Conscious Assistantscited by 36
  3. 2025Firewalls to Secure Dynamic LLM Agentic Networkscited by 34
  4. 2025DoomArena: A framework for Testing AI Agents Against Evolving Security Threatscited by 31
  5. 2025LLM Agents Should Employ Security Principlescited by 28
  6. 2026The Attack and Defense Landscape of Agentic AI: A Comprehensive Surveycited by 26
  7. 2026"Humans welcome to observe": A First Look at the Agent Social Network Moltbookcited by 25
  8. 2025Get Experience from Practice: LLM Agents with Record & Replaycited by 22
  9. 2025Towards Automating Data Access Permissions in AI Agentscited by 20
  10. 2026AgentLeak: A Benchmark for Internal-Channel Privacy Leakage in Multi-Agent LLM Systemscited by 20
  11. 2024Granular Privacy Control for Geolocation with Vision Language Modelscited by 19
  12. 2025Les Dissonances: Cross-Tool Harvesting and Polluting in Pool-of-Tools Empowered LLM Agentscited by 19
  13. 2025Searching for Privacy Risks in LLM Agents via Simulationcited by 19
  14. 2025The Sum Leaks More Than Its Parts: Compositional Privacy Risks and Mitigations in Multi-Agent Collaborationcited by 19
  15. 20251-2-3 Check: Enhancing Contextual Privacy in LLM via Multi-Agent Reasoningcited by 16
  16. 2025Position: Privacy Is Not Just Memorization!cited by 15
  17. 2025Privacy Reasoning in Ambiguous Contextscited by 11
  18. 2025SALT: Steering Activations towards Leakage-free Thinking in Chain of Thoughtcited by 11
  19. 2025Secure and Efficient Access Control for Computer-Use Agents via Context Spacecited by 10
  20. 2025Terrarium: Revisiting the Blackboard for Multi-Agent Safety, Privacy, and Security Studiescited by 10
  21. 2026AI Agents May Always Fall for Prompt Injectionscited by 9
  22. 2025Maris: A Formally Verifiable Privacy Policy Enforcement Paradigm for Multi-Agent Collaboration Systemscited by 8
  23. 2026Information-Theoretic Privacy Control for Sequential Multi-Agent LLM Systemscited by 8
  24. 2025Characterizing Unintended Consequences in Human-GUI Agent Collaboration for Web Browsingcited by 7
  25. 2025Beyond Jailbreaking: Auditing Contextual Privacy in LLM Agentscited by 7
  26. 2026The Promptware Kill Chain: How Prompt Injections Gradually Evolved Into a Multistep Malware Delivery Mechanismcited by 7
  27. 2026MemPrivacy: Privacy-Preserving Personalized Memory Management for Edge-Cloud Agentscited by 7
  28. 2024Privacy Bias in Language Models: A Contextual Integrity-based Auditing Metriccited by 6
  29. 2025Beyond Data Privacy: New Privacy Risks for Large Language Modelscited by 6
  30. 2026CalBench: Evaluating Coordination-Privacy Trade-offs in Multi-Agent LLMscited by 6
  31. 2025Privacy-Enhancing Paradigms within Federated Multi-Agent Systemscited by 5
  32. 2025Network-Level Prompt and Trait Leakage in Local Research Agentscited by 5
  33. 2026Towards trustworthy agentic AI: a comprehensive survey of safety, robustness, privacy, and system securitycited by 5
  34. 2026Toward Secure LLM Agents: Threat Surfaces, Attacks, Defenses, and Evaluationcited by 5
  35. 2025Can Large Language Models Really Recognize Your Name?cited by 4
  36. 2025Trusted AI Agents in the Cloudcited by 4
  37. 2026POLAR-Bench: A Diagnostic Benchmark for Privacy-Utility Trade-offs in LLM Agentscited by 4
  38. 2025Benchmarking LLM Privacy Recognition for Social Robot Decision Makingcited by 3

50 most cited of 90 All 90 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Protecting Users From Themselves: Safeguarding Contextual Privacy in Interactions with Conversational Agents
  2. 2026NeuroFilter: Activation-Based Guardrails for Privacy-Conscious LLM Agents
  3. 2025MAGPIE: A dataset for Multi-AGent contextual PrIvacy Evaluation
  4. 2026Agents That Know Too Much: A Data-Centric Survey of Privacy in LLM Agents
  5. 2026ADAM: A Systematic Data Extraction Attack on Agent Memory via Adaptive Querying
  6. 2025ConVerse: Benchmarking Contextual Safety in Agent-to-Agent Conversations
  7. 2026MuPPET: A Benchmark for Contextual Privacy of LLM Assistants in Multi-Party Conversations
  8. 2023"It's a Fair Game", or Is It? Examining How Users Navigate Disclosure Risks and Benefits When Using LLM-Based Conversational Agents