When "Correct" Is Not Safe: Can We Trust Functionally Correct Patches Generated by Code Agents?
On the map: LLM agents and tool use → Agent threats, guardrails and governance
Abstract (arXiv)
Code agents are increasingly trusted to autonomously fix bugs on platforms such as GitHub, yet their security evaluation focuses almost exclusively on functional correctness. In this paper, we reveal a novel type of threat to real-world code agents: Functionally Correct yet Vulnerable (FCV) patches, which pass all test cases but contain vulnerable code. With our proposed FCV-Attack, which can be deliberately crafted by malicious attackers or implicitly introduced by benign developers, we show that SOTA LLMs (e.g., ChatGPT and Claude) and agent scaffolds (e.g., SWE-agent and OpenHands) are all vulnerable to this FCV threat; across 12 agent-model combinations on SWE-Bench, the attack only requires black-box access and a single query to the code agent to perform the attack. For example, for CWE-538 (information exposure vulnerability), the FCV-Attack attains an attack success rate of 40.7% on GPT-5 Mini + OpenHands. Our results reveal an important security threat overlooked by current evaluation paradigms and urge the development of security-aware defenses for code agents.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use. Builds on 39 of its 49 references that are in the map. First posted on arXiv 15 Oct 2025.
Builds on
39 of 49 references.
Closest ancestors · 14
Same direction, newest first
- 2025Unintended Misalignment from Agentic Fine-Tuning: Risks and Mitigation19 Aug
- 2025Enhancing LLM Agent Safety via Causal Influence Prompting1 Jul
- 2025SEC-bench: Automated Benchmarking of LLM Agents on Real-World Software Security Tasks13 Jun
- 2025Think Twice Before You Act: Enhancing Agent Behavioral Safety with Thought Correction16 May
- 2025LlamaFirewall: An open source guardrail system for building secure AI agents6 May
- 2025AgentSpec: Customizable Runtime Enforcement for Safe and Reliable LLM Agents24 Mar
- 2025CVE-Bench: A Benchmark for AI Agents' Ability to Exploit Real-World Web Application Vulnerabilities21 Mar
- 2025Prompt Flow Integrity to Prevent Privilege Escalation in LLM Agents17 Mar
- 2025SafeArena: Evaluating the Safety of Autonomous Web Agents6 Mar
- 2025DemonAgent: Dynamically Encrypted Multi-Backdoor Implantation Attack on LLM-based Agent18 Feb
- 2024RedCode: Risky Code Execution and Generation Benchmark for Code Agents12 Nov
- 2024Teams of LLM Agents can Exploit Zero-Day Vulnerabilities2 Jun
2 more
Other references · 25
- 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
- 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
- 2024SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineeringcited by 1,628
- 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,326
- 2024OpenHands: An Open Platform for AI Software Developers as Generalist Agentscited by 956
- 2023AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Modelscited by 738
- 2024Agentless: Demystifying LLM-based Software Engineering Agentscited by 400
- 2024Training Software Engineering Agents and Verifiers with SWE-Gymcited by 300
- 2025SWE-RL: Advancing LLM Reasoning via Reinforcement Learning on Open Software Evolutioncited by 246
- 2023Purple Llama CyberSecEval: A Secure Coding Benchmark for Language Modelscited by 130
- 2024AgentScope: A Flexible yet Robust Multi-Agent Platformcited by 120
- 2025Trae Agent: An LLM-based Agent for Software Engineering with Test-time Scalingcited by 93
13 more
- 2025A Self-Improving Coding Agentcited by 58
- 2023CodeLMSec Benchmark: Systematically Evaluating and Finding Security Vulnerabilities in Black-Box Code Language Modelscited by 52
- 2025BaxBench: Can LLMs Generate Correct and Secure Backends?cited by 38
- 2025AutoRedTeamer: Autonomous Red Teaming with Lifelong Attack Integrationcited by 38
- 2025Evaluating Agent-based Program Repair at Googlecited by 35
- 2024CodeAgent: Autonomous Communicative Agents for Code Reviewcited by 33
- 2023DeceptPrompt: Exploiting LLM-driven Code Generation via Adversarial Natural Language Instructionscited by 31
- 2025Agentic AI Software Engineers: Programming with Trustcited by 19
- 2025SafeGenBench: A Benchmark Framework for Security Vulnerability Detection in LLM-Generated Codecited by 17
- 2025AgentScope 1.0: A Developer-Centric Framework for Building Agentic Applicationscited by 16
- 2025Benchmarking Prompt Engineering Techniques for Secure Code Generation with GPT Modelscited by 12
- 2025RedCoder: Automated Multi-Turn Red Teaming for Code LLMscited by 8
- 2025MOCHA: Are Code Language Models Robust Against Multi-Turn Malicious Coding Prompts?cited by 5
Built on by
Citing papers per year: 2026: 2.
2 papers.
Similar, not cited
Close in wording, no citation either way
- 2025SecureVibeBench: Benchmarking Secure Vibe Coding of AI Agents via Reconstructing Vulnerability-Introducing Scenarios
- 2026When Passing Tests Hides Vulnerabilities: An Empirical Study of Silent Failures in Agentic Systems
- 2025Red Teaming Program Repair Agents: When Correct Patches can Hide Vulnerabilities
- 2026MOSAIC-Bench: Measuring Compositional Vulnerability Induction in Coding Agents
- 2025How Safe Are AI-Generated Patches? A Large-scale Study on Security Risks in LLM and Agentic Automated Program Repair on SWE-bench
- 2024AutoSafeCoder: A Multi-Agent Framework for Securing LLM Code Generation through Static Analysis and Fuzz Testing
- 2024Fixing Security Vulnerabilities with AI in OSS-Fuzz
- 2026AXE: Grey-Box Exploitability Confirmation for Localized Vulnerability Reports