mlsanityData to 14 Oct 2026

Refused in Chat, Written in Code: Workflow-Level Jailbreak Construction in IDE Coding Agents

arXiv 2607.03968 ↗First posted 4 Jul 2026cs.SE cs.AIcited by 0

On the map: LLM agents and tool use → Agent threats, guardrails and governance

also in 1 more
Abstract (arXiv)

Large language models are increasingly deployed as IDE-integrated coding agents that decompose tasks, generate and edit files, run code, and refine outputs over many turns. Yet their safety is still often evaluated as if they were chatbots: one harmful prompt, one response, judged in isolation. We introduce workflow-level jailbreak construction, a failure mode in which a harmful objective is assembled across ordinary stages of a software-development workflow rather than generated through a single direct prompt. Using GitHub Copilot in Visual Studio Code, we study four closed-weight backends: Claude Sonnet 4.6, Claude Haiku 4.5, Gemini 3.1 Pro, and Gemini 3.5 Flash. Across 204 prompts from Hammurabi's Code, HarmBench, and AdvBench , the models show near-complete refusal under direct chat, CSV-read, and single-step code-fix baselines, with only 8/816 successful responses in each baseline condition. Under the full workflow, however, the same prompts and backends produce 816/816 unsafe teaching-shot completions, all independently confirmed by two expert evaluators under a strict rubric. These results show that conversational refusal benchmarks can substantially overstate the safety of deployed coding agents and motivate defenses that reason about safety across multi-turn IDE workflows and their generated artifacts, not only individual chat turns.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use. Builds on 29 of its 37 references that are in the map. First posted on arXiv 4 Jul 2026.

Builds on

29 of 37 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025Breaking the Code: Security Assessment of AI Code Agents Through Systematic Jailbreaking Attacks1 Oct
  2. 2024RedCode: Risky Code Execution and Generation Benchmark for Code Agents12 Nov
  3. 2024Refusal-Trained LLMs Are Easily Jailbroken As Browser Agents11 Oct
  4. 2024JAILJUDGE: A Comprehensive Jailbreak Judge Benchmark with Multi-Agent Enhanced Explanation Evaluation Framework11 Oct
  5. 2024AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agents11 Oct

Other references · 24

  1. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  2. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,104
  3. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,429
  4. 2021CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generationcited by 744
  5. 2024LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methodscited by 465
  6. 2023Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrationscited by 431
  7. 2023Large Language Models for Software Engineering: A Systematic Literature Reviewcited by 399
  8. 2024Great, Now Write an Article About That: The Crescendo Multi-Turn LLM Jailbreak Attackcited by 390
  9. 2022A Systematic Evaluation of Large Language Models of Codecited by 363
  10. 2024Foundational Challenges in Assuring Alignment and Safety of Large Language Modelscited by 199
  11. 2024Large Language Models are Inconsistent and Biased Evaluatorscited by 153
  12. 2023Purple Llama CyberSecEval: A Secure Coding Benchmark for Language Modelscited by 138
12 more
  1. 2024CodeAttack: Revealing Safety Generalization Challenges of Large Language Models via Code Completioncited by 76
  2. 2024Don't Say No: Jailbreaking LLM by Suppressing Refusalcited by 56
  3. 2024Language Models for Code Completion: A Practical Evaluationcited by 39
  4. 2025LLMs Cannot Reliably Judge (Yet?): A Comprehensive Assessment on the Robustness of LLM-as-a-Judgecited by 37
  5. 2025Jailbreak-R1: Exploring the Jailbreak Capabilities of LLMs via Reinforcement Learningcited by 27
  6. 2026A Coin Flip for Safety: LLM Judges Fail to Reliably Measure Adversarial Robustnesscited by 20
  7. 2024Endless Jailbreaks with Bijection Learningcited by 19
  8. 2026SEMA: Simple yet Effective Learning for Multi-Turn Jailbreak Attackscited by 12
  9. 2025RedCoder: Automated Multi-Turn Red Teaming for Code LLMscited by 9
  10. 2025Code Red! On the Harmfulness of Applying Off-the-shelf Large Language Models to Programming Taskscited by 6
  11. 2025Smoke and Mirrors: Jailbreaking LLM-based Code Generation via Implicit Malicious Promptscited by 5
  12. 2026What Breaks When LLMs Code? Characterizing Operational Safety Failures of Agentic Code Assistantscited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Code as a Weapon: A Consensus-Labeled Prompt Bank for Measuring Coding-Model Compliance with Malicious-Code Requests
  2. 2026Refusal Evaluation in Coding LLMs and Code Agents: A Systematic Review of Thirteen Malicious-Code Prompt Corpora (2023-2025)
  3. 2026CodeMimicry: Exploiting Safety Generalization Lag in Large Language Models via Structured Code Completion
  4. 2026A Validated Prompt Bank for Malicious Code Generation: Separating Executable Weapons from Security Knowledge in 1,554 Consensus-Labeled Prompts
  5. 2026SpecCloak: Probing Harmful Code Generation through Engineering Specifications
  6. 2026Prompt Injection Attacks on Agentic Coding Assistants: A Systematic Analysis of Vulnerabilities in Skills, Tools, and Protocol Ecosystems
  7. 2026Beyond Refusal: A Same-Lineage Study of Aligned and Abliterated LLMs for Vulnerability Analysis
  8. 2026MOSAIC-Bench: Measuring Compositional Vulnerability Induction in Coding Agents