Labels Are Not Endpoints: Treatment Leakage and Construct Validity in MCP Agent Security Evaluation
Not on the map. Why?
Abstract (arXiv)
Security evaluations of tool-using agents often equate stored labels with behavioral facts. We audit a preserved campaign by tracing 10,200 execution rows to 180 model-bound requests, 45 semantic requests, and 15 observable stimuli. Two schema treatments were delivered, but the planned external payload-family corpus was not. The historical grader exhibited direct treatment leakage: treatment metadata gated the ATTACK_SUCCESS class, so fixed behavior could change class under treatment relabeling. A treatment-blind reconstruction corrects 58 historical ATTACK_SUCCESS or HIJACK_ATTEMPT labels to authorized benign completions while preserving three verified protected-data transfers and one separate unauthorized-forwarding case. The locked v2 census contains exactly zero ATTACK_SUCCESS records, while the forwarding case remains a HIJACK_ATTEMPT at a semantic boundary concerning objective completion. A dual-reviewer blinded concordance review of all 96 requests deemed structurally interpretable by locked v2 produced identical reviewer-consensus classes but differed from the locked codebook on four construct-boundary cases. We contribute a seven-link Integrity Chain and an executable, scope-bounded endpoint-integrity linter. The result is a campaign-bounded measurement audit, not a population attack-rate, model-ranking, defense-efficacy, or causal estimate.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 13 of its 32 references that are in the map. First posted on arXiv 13 Aug 2026.
Builds on
13 of 32 references.
References · 13
- 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injectioncited by 1,443
- 2024InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agentscited by 538
- 2024Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agentscited by 369
- 2025Measuring what Matters: Construct Validity in Large Language Model Benchmarkscited by 119
- 2020Metamorphic Testing: A New Approach for Generating Next Test Casescited by 75
- 2025MCP Security Bench (MSB): Benchmarking Attacks Against Model Context Protocol in LLM Agentscited by 42
- 2025MCP-SafetyBench: A Benchmark for Safety Evaluation of Large Language Models with Real-World MCP Serverscited by 34
- 2025MCP-Guard: A Multi-Stage Defense-in-Depth Framework for Securing Model Context Protocol in Agentic AIcited by 19
- 2026VIGIL: Defending LLM Agents Against Tool Stream Injection via Verify-Before-Commitcited by 17
- 2026MCP-ITP: An Automated Framework for Implicit Tool Poisoning in MCPcited by 14
- 2026How Adversarial Environments Mislead Agentic AI?cited by 5
- 2026SafeMCP: Proactive Power Regulation for LLM Agent Defense via Environment-Grounded Look-Ahead Reasoningcited by 3
1 more
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026AgentDrift: A Step-Labeled Benchmark of Injection-Hijacked LLM Agent Trajectories
- 2026AgentSecBench: Measuring Prompt Injection, Privacy Leakage, and Tool-Use Integrity in LLM Agents
- 2026Position: Retire the "Positive Backdoor" Label -- Secret Alignment Requires Strict and Systematic Evaluation
- 2026SafeClawBench: Separating Semantic, Audit-Evidence, and Sandbox Harm in Tool-Using LLM Agents
- 2026Public-Sharing Labels and Verbatim Field Egress in an MCP-to-A2A Agent Configuration: A Controlled Multi-Model Study
- 2026Silent Failures in Agentic Security Evaluation: A Validated Harness for Tool-Call Mediation Under Indirect Prompt Injection
- 2026Don't believe everything you read: Understanding and Measuring MCP Behavior under Misleading Tool Descriptions
- 2026ContainmentBench: Trace-Based Evaluation of Post-Exposure Containment in Tool-Using LLM Agents