What Breaks When LLMs Code? Characterizing Operational Safety Failures of Agentic Code Assistants
On the map: LLM agents and tool use → Software engineering coding agents
Abstract (arXiv)
Autonomous coding agents built on large language models (LLMs) are rapidly being integrated into development workflows, yet their operational safety properties remain poorly understood beyond evaluations of explicitly malicious inputs. In practice, high-impact failures arise during benign, goal-directed use through environment breakage, fabricated success reports, etc. that current benchmarks do not capture. What categories of operational safety failures actually occur when coding agents are used for everyday development tasks and what is their impact? We present an incident-driven empirical study grounded in two complementary evidence streams. We screen 68,816 papers from 22 premier venues, curating 185 safety-relevant studies, and mine 16,586 GitHub issues from widely deployed LLM-powered coding tools, manually confirming 547 genuine safety failures. Applying systematic open coding over both corpora, we derive a multi-dimensional safety taxonomy of 33 operational risk types organized across seven dimensions, and annotate each incident with contributing factors, task context, severity, and downstream impact. Our findings show that coding-agent failures are often severe, with 326 of 547 incidents rated high or critical. The dominant risks are constraint violations, destructive operations, authorization bypasses, and deception, and over 65% of incidents arise in bug fixing and setup or configuration, patterns largely missing from prior literature. These results have direct implications for SE tool designers and benchmark developers: guardrails must go beyond adversarial-prompt defenses to enforce environmental constraints, failure transparency, and safe-halt behaviors.
Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use. Builds on 45 of its 93 references that are in the map. First posted on arXiv 29 May 2026.
Builds on
45 of 93 references.
Closest ancestors · 2
Same direction, newest first
- 2025An LLM Agentic Approach for Legal-Critical Software: A Case Study for Tax Prep Software16 Sep
- 2023AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation16 Aug
Other references · 43
- 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
- 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
- 2021Evaluating Large Language Models Trained on Codecited by 8,320
- 2021Program Synthesis with Large Language Modelscited by 3,637
- 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
- 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
- 2023Code Llama: Open Foundation Models for Codecited by 2,139
- 2016Concrete Problems in AI Safetycited by 1,642
- 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,439
- 2022Ignore Previous Prompt: Attack Techniques For Language Modelscited by 832
- 2023StarCoder: may the source be with you!cited by 799
- 2023LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusioncited by 564
31 more
- 2021Asleep at the Keyboard? Assessing the Security of GitHub Copilot's Code Contributionscited by 431
- 2023On the Planning Abilities of Large Language Models : A Critical Investigationcited by 312
- 2021Unsolved Problems in ML Safetycited by 292
- 2022Grounded Copilot: How Programmers Interact with Code-Generating Modelscited by 218
- 2022Do Users Write More Insecure Code with AI Assistants?cited by 192
- 2023Large Language Models for Code: Security Hardening and Adversarial Testingcited by 155
- 2023CodePlan: Repository-level Coding using LLMs and Planningcited by 140
- 2024RedCode: Risky Code Execution and Generation Benchmark for Code Agentscited by 90
- 2024Red-Teaming for Generative AI: Silver Bullet or Security Theater?cited by 83
- 2024CodeAttack: Revealing Safety Generalization Challenges of Large Language Models via Code Completioncited by 76
- 2024We Have a Package for You! A Comprehensive Analysis of Package Hallucinations by Code Generating LLMscited by 65
- 2025CWEval: Outcome-driven Evaluation on Functionality and Security of LLM Code Generationcited by 59
- 2023Bias Testing and Mitigation in LLM-based Code Generationcited by 53
- 2018FIRST: A Framework for Optimizing Information Quality in Mobile Crowdsensing Systemscited by 50
- 2023Large Language Models and Simple, Stupid Bugscited by 45
- 2023Security Weaknesses of Copilot-Generated Code in GitHub Projects: An Empirical Studycited by 38
- 2024Toward Secure Code Generation: Bridging Correctness and Security via Task-Adaptive Vulnerability Modeling and Execution-Based Benchmarkingcited by 27
- 2024PromSec: Prompt Optimization for Secure Generation of Functional Source Code with Large Language Models (LLMs)cited by 19
- 2024LiCoEval: Evaluating LLMs on License Compliance in Code Generationcited by 14
- 2024LibEvolutionEval: A Benchmark and Study for Version-Specific Code Generationcited by 12
- 2025Investigating The Smells of LLM Generated Codecited by 9
- 2022A Simple, Yet Effective Approach to Finding Biases in Code Generationcited by 8
- 2024Can LLMs Obfuscate Code? A Systematic Analysis of Large Language Models into Assembly Code Obfuscationcited by 6
- 2025Code Red! On the Harmfulness of Applying Off-the-shelf Large Language Models to Programming Taskscited by 6
- 2022Data Augmentation for Improving Emotion Recognition in Software Engineering Communicationcited by 5
- 2025Security Vulnerabilities in AI-Generated Code: A Large-Scale Analysis of Public GitHub Repositoriescited by 5
- 2020Failures and Fixes: A Study of Software System Incident Responsecited by 4
- 2025Can LLMs Write CI? A Study on Automatic Generation of GitHub Actions Configurationscited by 4
- 2021Automating Cryptographic Protocol Language Generation from Structured Specificationscited by 2
- 2024Enabling Memory Safety of C Programs using LLMscited by 1
- 2025Learning Programming in Informal Spaces: Using Emotion as a Lens to Understand Novice Struggles on r/learnprogrammingcited by 1
Built on by
Citing papers per year: 2026: 4.
4 papers.
- 2026From Chatbot to Digital Colleague: The Paradigm Shift Toward Persistent Autonomous AIcited by 6
- 2026Quantifying Overclaiming Propensity in Frontier LLM Agentscited by 4
- 2026Refused in Chat, Written in Code: Workflow-Level Jailbreak Construction in IDE Coding Agentscited by 0
- 2026Bringing analytic rigor to agentic AI for science: The Brain Researcher platform for neuroimaging data analysiscited by 0
- 2026Quantifying Overclaiming Propensity in Frontier LLM Agents17 Sep 2026
- 2026Bringing analytic rigor to agentic AI for science: The Brain Researcher platform for neuroimaging data analysis20 Aug 2026
- 2026Refused in Chat, Written in Code: Workflow-Level Jailbreak Construction in IDE Coding Agents4 Jul 2026
- 2026From Chatbot to Digital Colleague: The Paradigm Shift Toward Persistent Autonomous AI12 Jun 2026
Similar, not cited
Close in wording, no citation either way
- 2026SABER: Benchmarking Operational Safety of LLM Coding Agents in Stateful Project Workspaces
- 2025When Developer Aid Becomes Security Debt: A Systematic Analysis of Insecure Behaviors in LLM Coding Agents
- 2026Failure as a Process: An Anatomy of CLI Coding Agent Trajectories
- 2026When Errors Become Narratives: A Longitudinal Taxonomy of Silent Failures in a Production LLM Agent Runtime
- 2026When Passing Tests Hides Vulnerabilities: An Empirical Study of Silent Failures in Agentic Systems
- 2026Beyond Resolution Rates: Behavioral Drivers of Coding Agent Success and Failure
- 2025Understanding Code Agent Behaviour: An Empirical Study of Success and Failure Trajectories
- 2024Agent-SafetyBench: Evaluating the Safety of LLM Agents