MOSAIC-Bench: Measuring Compositional Vulnerability Induction in Coding Agents
On the map: LLM agents and tool use → Agent threats, guardrails and governance
Abstract (arXiv)
Coding agents often pass per-prompt safety review yet ship exploitable code when their tasks are decomposed into routine engineering tickets. The challenge is structural: existing safety alignment evaluates overt requests in isolation, leaving models blind to malicious end-states that emerge from sequenced compliance with innocuous-looking requests. We introduce MOSAIC-Bench (Malicious Objectives Sequenced As Innocuous Compliance), a benchmark of 199 three-stage attack chains paired with deterministic exploit oracles on deployed software substrates (10 web-application substrates, 31 CWE classes, 5 programming languages) that treats both exploit ground truth and downstream reviewer protocol as first-class evaluation axes. On this benchmark, nine production coding agents from Anthropic, OpenAI, Google, Moonshot, Zhipu, and Minimax compose innocuous tickets at 53-86% end-to-end ASR with only two refusals across all staged runs. In a matched direct-prompt experiment over four frontier Claude/Codex agents, vulnerable-output rates fall to 0-20.4%: Claude primarily refuses, while Codex primarily hardens rather than emitting the vulnerable implementation - ticket staging silences both defense modes simultaneously. Downstream, code reviewer agents approve 25.8% of these confirmed-vulnerable cumulative diffs as routine PRs, and a full-context implementation protocol closes only 50% of the staged/direct gap, ruling out context fragmentation as the sole explanation. As a deployable but non-adaptive mitigation, reframing the reviewer as an adversarial pentester reduces evasion across the evaluated reviewer subset; pentester framed evasion ranges from 3.0% to 17.6%, and an open-weight Gemma-4-E4B-it reviewer under this framing detects 88.4% of attacks on the dataset with a 4.6% false-positive rate measured on 608 real-world GitHub PRs.
Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use. Builds on 15 of its 19 references that are in the map. First posted on arXiv 5 May 2026.
Builds on
15 of 19 references.
Closest ancestors · 6
Same direction, newest first
- 2025Is Vibe Coding Safe? Benchmarking Vulnerability of Agent-Generated Code in Real-World Tasks2 Dec
- 2025Breaking the Code: Security Assessment of AI Code Agents Through Systematic Jailbreaking Attacks1 Oct
- 2025OpenAgentSafety: A Comprehensive Framework for Evaluating Real-World AI Agent Safety8 Jul
- 2025OS-Harm: A Benchmark for Measuring Safety of Computer Use Agents17 Jun
- 2025SEC-bench: Automated Benchmarking of LLM Agents on Real-World Software Security Tasks13 Jun
- 2024RedCode: Risky Code Execution and Generation Benchmark for Code Agents12 Nov
Other references · 9
- 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
- 2024Cybench: A Framework for Evaluating Cybersecurity Capabilities and Risks of Language Modelscited by 164
- 2024DrAttack: Prompt Decomposition and Reconstruction Makes Powerful LLM Jailbreakerscited by 101
- 2025BaxBench: Can LLMs Generate Correct and Secure Backends?cited by 41
- 2025Monitoring Decomposition Attacks in LLMs with Lightweight Sequential Monitorscited by 25
- 2025MOCHA: Are Code Language Models Robust Against Multi-Turn Malicious Coding Prompts?cited by 7
- 2026ABC-Bench: Benchmarking Agentic Backend Coding in Real-World Developmentcited by 6
- 2025VADER: A Human-Evaluated Benchmark for Vulnerability Assessment, Detection, Explanation, and Remediationcited by 5
- 2024SeCodePLT: A Unified Platform for Evaluating the Security of Code GenAIcited by 4
Built on by
Citing papers per year: 2026: 3.
3 papers.
- 2026Isolation as a First-Class Principle for LLM-Agent System Safety: Concepts, Taxonomy, Challenges and Future Directionscited by 1
- 2026Magnet: Detecting Cross-Session AI Misuse Through Capability Accumulationcited by 0
- 2026BulkPR-Bench: Benchmarking Queue-Level Governance of Interacting Pull Requestscited by 0
- 2026BulkPR-Bench: Benchmarking Queue-Level Governance of Interacting Pull Requests3 Aug 2026
- 2026Magnet: Detecting Cross-Session AI Misuse Through Capability Accumulation3 Aug 2026
- 2026Isolation as a First-Class Principle for LLM-Agent System Safety: Concepts, Taxonomy, Challenges and Future Directions14 Jul 2026
Similar, not cited
Close in wording, no citation either way
- 2026Beyond the Payload: How User Invocation Shapes Coding Agent Vulnerability to Repository Poisoning
- 2025SecureVibeBench: Benchmarking Secure Vibe Coding of AI Agents via Reconstructing Vulnerability-Introducing Scenarios
- 2026Trust but Verify? Uncovering the Security Debt of Autonomous Coding Agents
- 2026The Balkanization of Execution-Security Research for AI Coding Agents: Isolation, Access Control, and Time-of-Check-to-Time-of-Use Vulnerabilities
- 2026The Vulnerability With No CVE: Managing Persistent Gaps Between Mandate and Authority in AI Coding Agents
- 2026SecProbe: Adaptive Evaluation of Coding Agents on Cybersecurity Vulnerabilities
- 2025Red-Teaming Coding Agents from a Tool-Invocation Perspective: An Empirical Security Assessment
- 2026Towards a Risk Assessment of Malicious Skill Files in Coding Agents