mlsanityData to 14 Oct 2026

We Should Identify and Mitigate Third-Party Safety Risks in MCP-Powered Agent Systems

arXiv 2506.13666 ↗First posted 16 Jun 2025cs.LG cs.AIcited by 27

On the map: LLM agents and tool use → Agent threats, guardrails and governance

also in 1 more
Abstract (arXiv)

The development of large language models (LLMs) has entered in a experience-driven era, flagged by the emergence of environment feedback-driven learning via reinforcement learning and tool-using agents. This encourages the emergenece of model context protocol (MCP), which defines the standard on how should a LLM interact with external services, such as api and data. However, as MCP becomes the de facto standard for LLM agent systems, it also introduces new safety risks. In particular, MCP introduces third-party services, which are not controlled by the LLM developers, into the agent systems. These third-party MCP services provider are potentially malicious and have the economic incentives to exploit vulnerabilities and sabotage user-agent interactions. In this position paper, we advocate the research community in LLM safety to pay close attention to the new safety risks issues introduced by MCP, and develop new techniques to build safe MCP-powered agent systems. To establish our position, we argue with three key parts. (1) We first construct framework, a controlled framework to examine safety issues in MCP-powered agent systems. (2) We then conduct a series of pilot experiments to demonstrate the safety risks in MCP-powered agent systems is a real threat and its defense is not trivial. (3) Finally, we give our outlook by showing a roadmap to build safe MCP-powered agent systems. In particular, we would call for researchers to persue the following research directions: red teaming, MCP safe LLM development, MCP safety evaluation, MCP safety data accumulation, MCP service safeguard, and MCP safe ecosystem construction. We hope this position paper can raise the awareness of the research community in MCP safety and encourage more researchers to join this important research direction. Our code is available at https://github.com/littlelittlenine/SafeMCP.git.

Key facts. Cited by 27 papers in the map (data through 14 Oct 2026). Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use. Builds on 83 of its 103 references that are in the map. First posted on arXiv 16 Jun 2025.

Builds on

83 of 103 references.

Closest ancestors · 16

Same direction, newest first

  1. 2025A survey of agent interoperability protocols: Model Context Protocol (MCP), Agent Communication Protocol (ACP), Agent-to-Agent Protocol (A2A), and Agent Network Protocol (ANP)4 May
  2. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deployment22 Apr
  3. 2025MCP Guardian: A Security-First Layer for Safeguarding MCP-Based AI System17 Apr
  4. 2025MCP Safety Audit: LLMs with the Model Context Protocol Allow Major Security Exploits2 Apr
  5. 2025A Survey on Trustworthy LLM Agents: Threats and Countermeasures12 Mar
  6. 2025AgentSafe: Safeguarding Large Language Model-based Multi-agent Systems via Hierarchical Data Management6 Mar
  7. 2025Red-Teaming LLM Multi-Agent Systems via Communication Attacks20 Feb
  8. 2025G-Safeguard: A Topology-Guided Security Lens and Treatment on LLM-based Multi-agent Systems16 Feb
  9. 2025Commercial LLM Agents Are Already Vulnerable to Simple Yet Dangerous Attacks12 Feb
  10. 2024Agent-SafetyBench: Evaluating the Safety of LLM Agents19 Dec
  11. 2024From Allies to Adversaries: Manipulating LLM Tool-Calling through Adversarial Injection13 Dec
  12. 2024NetSafe: Exploring the Topological Safety of Multi-agent Networks21 Oct
4 more
  1. 2024Imprompter: Tricking LLM Agents into Improper Tool Use19 Oct
  2. 2024Breaking Agents: Compromising Autonomous LLM Agents Through Malfunction Amplification30 Jul
  3. 2024AgentPoison: Red-teaming LLM Agents via Poisoning Memory or Knowledge Bases17 Jul
  4. 2024Flooding Spread of Manipulated Knowledge in LLM-Based Multi-Agent Communities10 Jul

Other references · 67

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2023GPT-4 Technical Reportcited by 18,396
  4. 2022Training language models to follow instructions with human feedbackcited by 15,971
  5. 2025Qwen3 Technical Reportcited by 8,820
  6. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  7. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
  8. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  9. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  10. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,439
  11. 2023CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Societycited by 1,395
  12. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,251
38 more
  1. 2023A Survey on Evaluation of Large Language Modelscited by 1,242
  2. 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,207
  3. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,154
  4. 2023AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Modelscited by 785
  5. 2024Large Language Model based Multi-Agents: A Survey of Progress and Challengescited by 769
  6. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 729
  7. 2023A Comprehensive Overview of Large Language Modelscited by 648
  8. 2024A Survey on the Memory Mechanism of Large Language Model based Agentscited by 634
  9. 2023Textbooks Are All You Needcited by 569
  10. 2023MemoryBank: Enhancing Large Language Models with Long-Term Memorycited by 556
  11. 2024JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Modelscited by 536
  12. 2023Jailbreaking ChatGPT via Prompt Engineering: An Empirical Studycited by 498
  13. 2024Safety Alignment Should Be Made More Than Just a Few Tokens Deepcited by 472
  14. 2025Multi-Agent Collaboration Mechanisms: A Survey of LLMscited by 431
  15. 2023GPT-4 Is Too Smart To Be Safe: Stealthy Chat with LLMs via Ciphercited by 423
  16. 2023ToolAlpaca: Generalized Tool Learning for Language Models with 3000 Simulated Casescited by 393
  17. 2023Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructionscited by 387
  18. 2023DeepInception: Hypnotize Large Language Model to Be Jailbreakercited by 367
  19. 2018TextWorld: A Learning Environment for Text-based Gamescited by 310
  20. 2024PoisonedRAG: Knowledge Corruption Attacks to Retrieval-Augmented Generation of Large Language Modelscited by 267
  21. 2025Model Context Protocol (MCP): Landscape, Security Threats, and Future Research Directionscited by 265
  22. 2023A Wolf in Sheep's Clothing: Generalized Nested Jailbreak Prompts can Fool Large Language Models Easilycited by 259
  23. 2023The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learningcited by 257
  24. 2023GPT4Tools: Teaching Large Language Model to Use Tools via Self-instructioncited by 248
  25. 2024ToolACE: Winning the Points of LLM Function Callingcited by 216
  26. 2024Foundational Challenges in Assuring Alignment and Safety of Large Language Modelscited by 199
  27. 2023Augmenting Language Models with Long-Term Memorycited by 189
  28. 2025Multi-agent Architecture Search via Agentic Supernetcited by 179
  29. 2023ADaPT: As-Needed Decomposition and Planning with Language Modelscited by 177
  30. 2024Safety Fine-Tuning at (Almost) No Cost: A Baseline for Vision Large Language Modelscited by 167
  31. 2023Poisoning Retrieval Corpora by Injecting Adversarial Passagescited by 159
  32. 2024The Good and The Bad: Exploring Privacy Issues in Retrieval-Augmented Generation (RAG)cited by 159
  33. 2024Certifiably Robust RAG against Retrieval Corruptioncited by 151
  34. 2024Cut the Crap: An Economical Communication Pipeline for LLM-based Multi-Agent Systemscited by 141
  35. 2025SafeChain: Safety of Language Models with Long Chain-of-Thought Reasoning Capabilitiescited by 136
  36. 2024AI Agents Under Threat: A Survey of Key Security Challenges and Future Pathwayscited by 127
  37. 2024AgentGym: Evolving Large Language Model-based Agents across Diverse Environmentscited by 120
  38. 2024G-Designer: Architecting Multi-agent Communication Topologies via Graph Neural Networkscited by 119

50 most cited of 67 All 67 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 18; 2026: 9.

27 papers.

Sort:

By area:

  1. 2025A Survey of Context Engineering for Large Language Modelscited by 126
  2. 2025A Survey of LLM-Driven AI Agent Communication: Protocols, Security Risks, and Defense Countermeasurescited by 57
  3. 2025MCPSecBench: A Systematic Security Benchmark and Playground for Testing Model Context Protocolscited by 51
  4. 2025MCP Security Bench (MSB): Benchmarking Attacks Against Model Context Protocol in LLM Agentscited by 42
  5. 2025MCP-SafetyBench: A Benchmark for Safety Evaluation of Large Language Models with Real-World MCP Serverscited by 34
  6. 2025AgentBound: Securing Execution Boundaries of AI Agentscited by 23
  7. 2025Aegis: Automated Error Generation and Attribution for Multi-Agent Systemscited by 22
  8. 2025MCP-Guard: A Multi-Stage Defense-in-Depth Framework for Securing Model Context Protocol in Agentic AIcited by 19
  9. 2025Automatic Red Teaming LLM-based Agents with Model Context Protocol Toolscited by 18
  10. 2025MindGuard: Intrinsic Decision Inspection for Securing LLM Agents Against Metadata Poisoningcited by 10
  11. 2026BalDRO: A Distributionally Robust Optimization based Framework for Large Language Model Unlearningcited by 8
  12. 2025A Large-Scale Evolvable Dataset for Model Context Protocol Ecosystem and Security Analysiscited by 7
15 more
  1. 2025Beyond Data Privacy: New Privacy Risks for Large Language Modelscited by 6
  2. 2025Agentic Services Computingcited by 6
  3. 2025SafeSearch: Automated Red-Teaming of LLM-Based Search Agentscited by 4
  4. 2026MCPShield: A Security Cognition Layer for Adaptive Trust Calibration in Model Context Protocol Agentscited by 3
  5. 2026SafeMCP: Proactive Power Regulation for LLM Agent Defense via Environment-Grounded Look-Ahead Reasoningcited by 3
  6. 2025Quantifying Conversation Drift in MCP via Latent Polytopecited by 2
  7. 2026Invisible Threats from Model Context Protocol: Generating Stealthy Injection Payload via Tree-based Adaptive Searchcited by 2
  8. 2026MCP-DPT: A Defense-Placement Taxonomy and Coverage Analysis for Model Context Protocol Securitycited by 2
  9. 2025Model Context Protocol for Vision Systems: Audit, Security, and Protocol Extensionscited by 1
  10. 2025Can LLM Infer Risk Information From MCP Server System Logs?cited by 1
  11. 2026From Secure Agentic AI to Secure Agentic Web: Challenges, Threats, and Future Directionscited by 1
  12. 2026MCP-in-SoS: Risk assessment framework for open-source MCP serverscited by 1
  13. 2025Servant, Stalker, Predator: How An Honest, Helpful, And Harmless (3H) Agent Unlocks Adversarial Skillscited by 0
  14. 2026TrustShiftProbe: Characterizing, Benchmarking, and Defending Staged Trust Attacks on MCP Serverscited by 0
  15. 2026Connecting the Dots in Agentic AI Security: A Cross-Dimensional Threat Taxonomy, Evaluation Maturity, and Open Challengescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Beyond Single-Agent Safety: A Taxonomy of Risks in LLM-to-LLM Interactions
  2. 2026Towards Verifiably Safe Tool Use for LLM Agents
  3. 2026Hybrid Analysis for Secure MCP Tool Use in LLM Agents
  4. 2025Beyond Reactive Safety: Risk-Aware LLM Alignment via Long-Horizon Simulation
  5. 2026S3: Improving Agent Safety through Multi-Stage Defense
  6. 2026Risky-Bench: Probing Agentic Safety Risks under Real-World Deployment
  7. 2025Agent Safety Alignment via Reinforcement Learning
  8. 2025Survey of LLM Agent Communication with MCP: A Software Design Pattern Centric Review