SoK: AI Secure Code Generation: Progress, Pitfalls, and Paths Forward
On the map: LLM agents and tool use → Agent threats, guardrails and governance
Abstract (arXiv)
The increasing use of AI systems for code generation raises a central security question: what can today's models and coding agents actually do to produce secure code, where do they still fail, and what would move the field forward? Existing work has explored prompting, fine-tuning, reinforcement learning, and agentic workflows for secure code generation, but the field still lacks a systematic understanding of how these techniques improve security and why substantial failures persist. In this SoK, we systematize the progress, pitfalls, and paths forward for AI secure code generation. We introduce a three-level framework that measures models' natural-language understanding of secure coding principles, their code-level actuation of those principles during generation, and the knowledge--actuation gaps between the two. We instantiate this framework across models and coding agents on benchmarks covering both isolated function-level security and full web-application security. Our results show that secure-coding-principle understanding is a statistically strong predictor of code-level outcomes, including functional correctness, security, and joint functional-security correctness. Yet substantial knowledge--actuation gaps remain: models can recognize relevant security principles but still fail to translate them into secure and functional code. These findings offer a principle-centered account of where AI secure code generation stands today and identify concrete paths forward through principle-guided generation, evaluation, benchmarking, and agentic workflows.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use. Builds on 44 of its 57 references that are in the map. First posted on arXiv 23 Jun 2026.
Builds on
44 of 57 references.
Closest ancestors · 4
Same direction, newest first
- 2025SEC-bench: Automated Benchmarking of LLM Agents on Real-World Software Security Tasks13 Jun
- 2025SCGAgent: Recreating the Benefits of Reasoning Models for Secure Code Generation with Agentic Workflows8 Jun
- 2025SecRepoBench: Benchmarking Code Agents for Secure Code Completion in Real-World Repositories29 Apr
- 2024AutoSafeCoder: A Multi-Agent Framework for Securing LLM Code Generation through Static Analysis and Fuzz Testing16 Sep
Other references · 40
- 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
- 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682
- 2024SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineeringcited by 1,737
- 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,729
- 2021Asleep at the Keyboard? Assessing the Security of GitHub Copilot's Code Contributionscited by 431
- 2023Language Models can Solve Computer Taskscited by 372
- 2019Cosmos QA: Machine Reading Comprehension with Contextual Commonsense Reasoningcited by 285
- 2020ReClor: A Reading Comprehension Dataset Requiring Logical Reasoningcited by 217
- 2022Do Users Write More Insecure Code with AI Assistants?cited by 192
- 2023Large Language Models for Code: Security Hardening and Adversarial Testingcited by 155
- 2023Purple Llama CyberSecEval: A Secure Coding Benchmark for Language Modelscited by 138
- 2022Lost at C: A User Study on the Security Implications of Large Language Model Code Assistantscited by 91
28 more
- 2024CyberSecEval 2: A Wide-Ranging Cybersecurity Evaluation Suite for Large Language Modelscited by 89
- 2023How Secure is Code Generated by ChatGPT?cited by 81
- 2024Instruction Tuning for Secure Code Generationcited by 68
- 2023LLMSecEval: A Dataset of Natural Language Prompts for Security Evaluationscited by 62
- 2025CWEval: Outcome-driven Evaluation on Functionality and Security of LLM Code Generationcited by 59
- 2023CodeLMSec Benchmark: Systematically Evaluating and Finding Security Vulnerabilities in Black-Box Code Language Modelscited by 55
- 2024Large Language Model for Vulnerability Detection and Repair: Literature Review and the Road Aheadcited by 42
- 2025BaxBench: Can LLMs Generate Correct and Secure Backends?cited by 41
- 2023SALLM: Security Assessment of Generated Codecited by 39
- 2024Constrained Decoding for Secure Code Generationcited by 37
- 2024Prompting Techniques for Secure Code Generation: A Systematic Investigationcited by 34
- 2024Toward Secure Code Generation: Bridging Correctness and Security via Task-Adaptive Vulnerability Modeling and Execution-Based Benchmarkingcited by 27
- 2024ProSec: Fortifying Code LLMs with Proactive Security Alignmentcited by 25
- 2024PromSec: Prompt Optimization for Secure Generation of Functional Source Code with Large Language Models (LLMs)cited by 19
- 2025PurpCode: Reasoning for Safer Code Generationcited by 18
- 2024HexaCoder: Secure Code Generation via Oracle-Guided Synthetic Training Datacited by 16
- 2024SecCoder: Towards Generalizable and Robust Secure Code Generationcited by 14
- 2026RealSec-bench: A Benchmark for Evaluating Secure Code Generation in Real-World Repositoriescited by 14
- 2025SoK: Automated Vulnerability Repair: Methods, Tools, and Assessmentscited by 12
- 2025VulnRepairEval: An Exploit-Based Evaluation Framework for Assessing Large Language Model Vulnerability Repair Capabilitiescited by 12
- 2025SoK: Towards Effective Automated Vulnerability Repaircited by 10
- 2025Rethinking the Evaluation of Secure Code Generationcited by 10
- 2025Teaching an Old LLM Secure Coding: Localized Preference Optimization on Distilled Preferencescited by 10
- 2026ARVO: Atlas of Reproducible Vulnerabilities for Open-Source Softwarecited by 9
- 2025RESCUE: Retrieval Augmented Secure Code Generationcited by 7
- 2024Static Application Security Testing (SAST) Tools for Smart Contracts: How Far Are We?cited by 5
- 2024LLMSecCode: Evaluating Large Language Models for Secure Codingcited by 4
- 2026Detect--Repair--Verify for LLM-Generated Code: A Multi-Language, Multi-Granularity Empirical Studycited by 1
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026MACGen: Toward Functionally Correct and Secure Code Generation via Multi-Agent Collaboration
- 2026Security-by-Design for LLM-Based Code Generation: Leveraging Internal Representations for Concept-Driven Steering Mechanisms
- 2025DualGauge: Automated Joint Security-Functionality Benchmarking of Specification-Only Code Generation by LLMs and Coding Agents
- 2025SecureVibeBench: Benchmarking Secure Vibe Coding of AI Agents via Reconstructing Vulnerability-Introducing Scenarios
- 2026Secure Code Generation via Online Reinforcement Learning with Vulnerability Reward Model
- 2026Insights into Security-Related AI-Generated Pull Requests
- 2026Security in the Age of AI Teammates: An Empirical Study of Agentic Pull Requests on GitHub
- 2025GUARD:Dual-Agent based Backdoor Defense on Chain-of-Thought in Neural Code Generation