Coding with "Enemy": Can Human Developers Detect AI Agent Sabotage?
On the map: LLM agents and tool use → Software engineering coding agents
Abstract (arXiv)
AI coding agents are increasingly embedded in real-world software development, collaborating with human developers while gaining broader access to codebases and tools. This creates a new attack surface: an agent can exploit human trust to sabotage development, for instance by inserting malicious code to accomplish a hidden side task. Most prior work studies AI sabotage in AI-only settings, paying limited attention to the role of human oversight in detecting and mitigating such malicious behavior. To address this gap, we conduct the first large-scale study of human oversight in AI coding sabotage. Over 100 participants collaborate with one of four frontier models (Claude-Opus-4.6, GPT-5.4, Gemini-3.1-Pro, and MiniMax-M2.7) on a long-horizon coding task lasting around five hours, designed to mimic real-world workflows. We find that 83/88 (94%) of developers in the no-monitor conditions fail to detect sabotage, and our analysis of participant feedback attributes this vulnerability to minimal code review, plausible cover story, and overtrust in agents. We further test the effectiveness of a safety monitor in one condition: while the monitor reduces sabotage success, sabotage still succeeds in 9/16 (56%) of sessions with a correct monitor alert. Drawing on participant feedback, we offer actionable suggestions for better monitor design. This work complements existing AI safety research and highlights an urgent need for human-centric safety mechanisms that account for human factors, particularly in long-horizon, real-world development settings.
Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use. Builds on 24 of its 36 references that are in the map. First posted on arXiv 4 Jun 2026.
Builds on
24 of 36 references.
Closest ancestors · 1
Same direction, newest first
Other references · 23
- 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
- 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environmentscited by 1,195
- 2026Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfacescited by 585
- 2024AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agentscited by 437
- 2024Alignment faking in large language modelscited by 360
- 2024Frontier Models are Capable of In-context Schemingcited by 287
- 2023AI Control: Improving Safety Despite Intentional Subversioncited by 229
- 2024ResearchAgent: Iterative Research Idea Generation over Scientific Literature with Large Language Modelscited by 210
- 2025Agentic Misalignment: How LLMs Could Be Insider Threatscited by 158
- 2025Natural Emergent Misalignment from Reward Hacking in Production RLcited by 136
- 2025CyberGym: Evaluating AI Agents' Real-World Cybersecurity Capabilities at Scalecited by 78
- 2025SHADE-Arena: Evaluating Sabotage and Monitoring in LLM Agentscited by 71
11 more
- 2024Sabotage Evaluations for Frontier Modelscited by 52
- 2026Agents of Chaoscited by 32
- 2025Reliable Weak-to-Strong Monitoring of LLM Agentscited by 31
- 2026Unsafer in Many Turns: Benchmarking and Defending Multi-Turn Safety Risks in Tool-Using Agentscited by 30
- 2025Async Control: Stress-testing Asynchronous Control Measures for LLM Agentscited by 22
- 2025Professional Software Developers Don't Vibe, They Control: AI Agent Use for Coding in 2025cited by 20
- 2025LLMs Can Covertly Sandbag on Capability Evaluations Against Chain-of-Thought Monitoringcited by 18
- 2025BashArena: A Control Setting for Highly Privileged AI Agentscited by 17
- 2025CTRL-ALT-DECEIT: Sabotage Evaluations for Automated AI R&Dcited by 15
- 2026How does information access affect LLM monitors' ability to detect sabotage?cited by 14
- 2025Optimizing AI Agent Attacks With Synthetic Datacited by 6
Built on by
Citing papers per year: 2026: 4.
4 papers.
- 2026SecureVibe: Making Vibe Coding More Secure29 Sep 2026
- 2026ParallelPilot: Supporting Coordination and Monitoring in Parallel AI Coding27 Sep 2026
- 2026AI Persuasion as a Threat to Human Control13 Sep 2026
- 2026ResearchArena: Evaluating Sabotage and Monitoring in Automated AI R&D21 Jul 2026
Similar, not cited
Close in wording, no citation either way
- 2026Security in the Age of AI Teammates: An Empirical Study of Agentic Pull Requests on GitHub
- 2026Human oversight of agentic systems in practice: Examining the oversight work, challenges, and heuristics of developers using software agents
- 2026Trust but Verify? Uncovering the Security Debt of Autonomous Coding Agents
- 2026ParanoiaEval: Benchmarking Unnecessary Defensive Work in Agentic Coding
- 2026Humans are Missing from AI Coding Agent Research
- 2026MOSAIC-Bench: Measuring Compositional Vulnerability Induction in Coding Agents
- 2026Towards a Risk Assessment of Malicious Skill Files in Coding Agents
- 2026Beyond the Payload: How User Invocation Shapes Coding Agent Vulnerability to Repository Poisoning