mlsanityData to 14 Oct 2026

What Breaks When LLMs Code? Characterizing Operational Safety Failures of Agentic Code Assistants

arXiv 2605.30777 ↗First posted 29 May 2026cs.SEcited by 4

On the map: LLM agents and tool use → Software engineering coding agents

Abstract (arXiv)

Autonomous coding agents built on large language models (LLMs) are rapidly being integrated into development workflows, yet their operational safety properties remain poorly understood beyond evaluations of explicitly malicious inputs. In practice, high-impact failures arise during benign, goal-directed use through environment breakage, fabricated success reports, etc. that current benchmarks do not capture. What categories of operational safety failures actually occur when coding agents are used for everyday development tasks and what is their impact? We present an incident-driven empirical study grounded in two complementary evidence streams. We screen 68,816 papers from 22 premier venues, curating 185 safety-relevant studies, and mine 16,586 GitHub issues from widely deployed LLM-powered coding tools, manually confirming 547 genuine safety failures. Applying systematic open coding over both corpora, we derive a multi-dimensional safety taxonomy of 33 operational risk types organized across seven dimensions, and annotate each incident with contributing factors, task context, severity, and downstream impact. Our findings show that coding-agent failures are often severe, with 326 of 547 incidents rated high or critical. The dominant risks are constraint violations, destructive operations, authorization bypasses, and deception, and over 65% of incidents arise in bug fixing and setup or configuration, patterns largely missing from prior literature. These results have direct implications for SE tool designers and benchmark developers: guardrails must go beyond adversarial-prompt defenses to enforce environmental constraints, failure transparency, and safe-halt behaviors.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use. Builds on 45 of its 93 references that are in the map. First posted on arXiv 29 May 2026.

Builds on

45 of 93 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025An LLM Agentic Approach for Legal-Critical Software: A Case Study for Tax Prep Software16 Sep
  2. 2023AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation16 Aug

Other references · 43

  1. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  4. 2021Program Synthesis with Large Language Modelscited by 3,637
  5. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  6. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
  7. 2023Code Llama: Open Foundation Models for Codecited by 2,139
  8. 2016Concrete Problems in AI Safetycited by 1,642
  9. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,439
  10. 2022Ignore Previous Prompt: Attack Techniques For Language Modelscited by 832
  11. 2023StarCoder: may the source be with you!cited by 799
  12. 2023LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusioncited by 564
31 more
  1. 2021Asleep at the Keyboard? Assessing the Security of GitHub Copilot's Code Contributionscited by 431
  2. 2023On the Planning Abilities of Large Language Models : A Critical Investigationcited by 312
  3. 2021Unsolved Problems in ML Safetycited by 292
  4. 2022Grounded Copilot: How Programmers Interact with Code-Generating Modelscited by 218
  5. 2022Do Users Write More Insecure Code with AI Assistants?cited by 192
  6. 2023Large Language Models for Code: Security Hardening and Adversarial Testingcited by 155
  7. 2023CodePlan: Repository-level Coding using LLMs and Planningcited by 140
  8. 2024RedCode: Risky Code Execution and Generation Benchmark for Code Agentscited by 90
  9. 2024Red-Teaming for Generative AI: Silver Bullet or Security Theater?cited by 83
  10. 2024CodeAttack: Revealing Safety Generalization Challenges of Large Language Models via Code Completioncited by 76
  11. 2024We Have a Package for You! A Comprehensive Analysis of Package Hallucinations by Code Generating LLMscited by 65
  12. 2025CWEval: Outcome-driven Evaluation on Functionality and Security of LLM Code Generationcited by 59
  13. 2023Bias Testing and Mitigation in LLM-based Code Generationcited by 53
  14. 2018FIRST: A Framework for Optimizing Information Quality in Mobile Crowdsensing Systemscited by 50
  15. 2023Large Language Models and Simple, Stupid Bugscited by 45
  16. 2023Security Weaknesses of Copilot-Generated Code in GitHub Projects: An Empirical Studycited by 38
  17. 2024Toward Secure Code Generation: Bridging Correctness and Security via Task-Adaptive Vulnerability Modeling and Execution-Based Benchmarkingcited by 27
  18. 2024PromSec: Prompt Optimization for Secure Generation of Functional Source Code with Large Language Models (LLMs)cited by 19
  19. 2024LiCoEval: Evaluating LLMs on License Compliance in Code Generationcited by 14
  20. 2024LibEvolutionEval: A Benchmark and Study for Version-Specific Code Generationcited by 12
  21. 2025Investigating The Smells of LLM Generated Codecited by 9
  22. 2022A Simple, Yet Effective Approach to Finding Biases in Code Generationcited by 8
  23. 2024Can LLMs Obfuscate Code? A Systematic Analysis of Large Language Models into Assembly Code Obfuscationcited by 6
  24. 2025Code Red! On the Harmfulness of Applying Off-the-shelf Large Language Models to Programming Taskscited by 6
  25. 2022Data Augmentation for Improving Emotion Recognition in Software Engineering Communicationcited by 5
  26. 2025Security Vulnerabilities in AI-Generated Code: A Large-Scale Analysis of Public GitHub Repositoriescited by 5
  27. 2020Failures and Fixes: A Study of Software System Incident Responsecited by 4
  28. 2025Can LLMs Write CI? A Study on Automatic Generation of GitHub Actions Configurationscited by 4
  29. 2021Automating Cryptographic Protocol Language Generation from Structured Specificationscited by 2
  30. 2024Enabling Memory Safety of C Programs using LLMscited by 1
  31. 2025Learning Programming in Informal Spaces: Using Emotion as a Lens to Understand Novice Struggles on r/learnprogrammingcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 4.

4 papers.

Similar, not cited

Close in wording, no citation either way

  1. 2026SABER: Benchmarking Operational Safety of LLM Coding Agents in Stateful Project Workspaces
  2. 2025When Developer Aid Becomes Security Debt: A Systematic Analysis of Insecure Behaviors in LLM Coding Agents
  3. 2026Failure as a Process: An Anatomy of CLI Coding Agent Trajectories
  4. 2026When Errors Become Narratives: A Longitudinal Taxonomy of Silent Failures in a Production LLM Agent Runtime
  5. 2026When Passing Tests Hides Vulnerabilities: An Empirical Study of Silent Failures in Agentic Systems
  6. 2026Beyond Resolution Rates: Behavioral Drivers of Coding Agent Success and Failure
  7. 2025Understanding Code Agent Behaviour: An Empirical Study of Success and Failure Trajectories
  8. 2024Agent-SafetyBench: Evaluating the Safety of LLM Agents