# Toward Secure Code Generation: Bridging Correctness and Security via Task-Adaptive Vulnerability Modeling and Execution-Based Benchmarking

arXiv: https://arxiv.org/abs/2407.02395 · HTML: https://mlsanity.com/paper/2407.02395/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 36 references that are in the map.
- First posted on arXiv 2 Jul 2024.

## Builds on (most cited first, 20 of 20)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Code Llama: Open Foundation Models for Code](https://mlsanity.com/paper/2308.12950/) (2023)
- [Measuring Coding Challenge Competence With APPS](https://mlsanity.com/paper/2105.09938/) (2021)
- [CodeBLEU: a Method for Automatic Evaluation of Code Synthesis](https://mlsanity.com/paper/2009.10297/) (2020)
- [DS-1000: A Natural and Reliable Benchmark for Data Science Code Generation](https://mlsanity.com/paper/2211.11501/) (2022)
- [Asleep at the Keyboard? Assessing the Security of GitHub Copilot's Code Contributions](https://mlsanity.com/paper/2108.09293/) (2021)
- [Do Users Write More Insecure Code with AI Assistants?](https://mlsanity.com/paper/2211.03622/) (2022)
- [Large Language Models for Code: Security Hardening and Adversarial Testing](https://mlsanity.com/paper/2302.05319/) (2023)
- [Purple Llama CyberSecEval: A Secure Coding Benchmark for Language Models](https://mlsanity.com/paper/2312.04724/) (2023)
- [Mapping Language to Code in Programmatic Context](https://mlsanity.com/paper/1808.09588/) (2018)
- [How Secure is Code Generated by ChatGPT?](https://mlsanity.com/paper/2304.09655/) (2023)
- [Instruction Tuning for Secure Code Generation](https://mlsanity.com/paper/2402.09497/) (2024)
- [Is GitHub's Copilot as Bad as Humans at Introducing Vulnerabilities in Code?](https://mlsanity.com/paper/2204.04741/) (2022)
- [LLMSecEval: A Dataset of Natural Language Prompts for Security Evaluations](https://mlsanity.com/paper/2303.09384/) (2023)
- [CodeLMSec Benchmark: Systematically Evaluating and Finding Security Vulnerabilities in Black-Box Code Language Models](https://mlsanity.com/paper/2302.04012/) (2023)
- [JuICe: A Large Scale Distantly Supervised Dataset for Open Domain Context-based Code Generation](https://mlsanity.com/paper/1910.02216/) (2019)
- [Fault-Aware Neural Code Rankers](https://mlsanity.com/paper/2206.03865/) (2022)
- [Automatic Static Bug Detection for Machine Learning Libraries: Are We There Yet?](https://mlsanity.com/paper/2307.04080/) (2023)

## Built on by (most cited first, 27 of 27)

- [Large Language Models for Code Generation: A Comprehensive Survey of Challenges, Techniques, Evaluation, and Applications](https://mlsanity.com/paper/2503.01245/) (2025)
- [The Scales of Justitia: A Comprehensive Survey on Safety Evaluation of LLMs](https://mlsanity.com/paper/2506.11094/) (2025)
- [Correctness Assessment of Code Generated by Large Language Models Using Internal Representations](https://mlsanity.com/paper/2501.12934/) (2025)
- [Large Language Models (LLMs) for Source Code Analysis: applications, models and datasets](https://mlsanity.com/paper/2503.17502/) (2025)
- [From Code Foundation Models to Agents and Applications: A Comprehensive Survey and Practical Guide to Code Intelligence](https://mlsanity.com/paper/2511.18538/) (2025)
- [Vulnerability Detection: From Formal Verification to Large Language Models and Hybrid Approaches: A Comprehensive Overview](https://mlsanity.com/paper/2503.10784/) (2025)
- [CodeMirage: A Multi-Lingual Benchmark for Detecting AI-Generated and Paraphrased Source Code from Production-Level LLMs](https://mlsanity.com/paper/2506.11059/) (2025)
- [AIGCodeSet: A New Annotated Dataset for AI Generated Code Detection](https://mlsanity.com/paper/2412.16594/) (2024)
- [LLM Benchmarking with LLaMA2: Evaluating Code Development Performance Across Multiple Programming Languages](https://mlsanity.com/paper/2503.19217/) (2025)
- [How Safe Are AI-Generated Patches? A Large-scale Study on Security Risks in LLM and Agentic Automated Program Repair on SWE-bench](https://mlsanity.com/paper/2507.02976/) (2025)
- [Prompt to Pwn: Automated Exploit Generation for Smart Contracts](https://mlsanity.com/paper/2508.01371/) (2025)
- [How Secure is Secure Code Generation? Adversarial Prompts Put LLM Defenses to the Test](https://mlsanity.com/paper/2601.07084/) (2026)
- [What Breaks When LLMs Code? Characterizing Operational Safety Failures of Agentic Code Assistants](https://mlsanity.com/paper/2605.30777/) (2026)
- [SOK: Exploring Hallucinations and Security Risks in AI-Assisted Software Development with Insights for LLM Deployment](https://mlsanity.com/paper/2502.18468/) (2025)
- [MultiAIGCD: A Comprehensive dataset for AI Generated Code Detection Covering Multiple Languages, Models,Prompts, and Scenarios](https://mlsanity.com/paper/2507.21693/) (2025)
- [RA-Gen: A Controllable Code Generation Framework Using ReAct for Multi-Agent Task Execution](https://mlsanity.com/paper/2510.08665/) (2025)
- [Automatic Detection of LLM-Generated Code: A Comparative Case Study of Contemporary Models Across Function and Class Granularities](https://mlsanity.com/paper/2409.01382/) (2024)
- [Program Semantic Inequivalence Game with Large Language Models](https://mlsanity.com/paper/2505.03818/) (2025)
- [LLM Performance for Code Generation on Noisy Tasks](https://mlsanity.com/paper/2505.23598/) (2025)
- [Optimizing Token Choice for Code Watermarking: An RL Approach](https://mlsanity.com/paper/2508.11925/) (2025)
- [Secure Code Generation at Scale with Reflexion](https://mlsanity.com/paper/2511.03898/) (2025)
- [Extracting Recurring Vulnerabilities from Black-Box LLM-Generated Software](https://mlsanity.com/paper/2602.04894/) (2026)
- [Refusal Evaluation in Coding LLMs and Code Agents: A Systematic Review of Thirteen Malicious-Code Prompt Corpora (2023-2025)](https://mlsanity.com/paper/2605.20351/) (2026)
- [GitHub's Copilot Code Review: Can AI Spot Security Flaws Before You Commit?](https://mlsanity.com/paper/2509.13650/) (2025)
- [Is Vibe Coding the Future? An Empirical Assessment of LLM Generated Codes for Construction Safety](https://mlsanity.com/paper/2604.12311/) (2026)
- [PYTHALAB-MERA: Validation-Grounded Memory, Retrieval, and Acceptance Control for Frozen-LLM Coding Agents](https://mlsanity.com/paper/2605.08468/) (2026)
- [SoK: AI Secure Code Generation: Progress, Pitfalls, and Paths Forward](https://mlsanity.com/paper/2606.25195/) (2026)

## Abstract (arXiv)

Large language models (LLMs) are increasingly used for program synthesis, yet they often generate code that is functionally plausible but insecure. Progress in secure code generation has been hindered by benchmarks that are small, non-executable, leak mitigation details, or rely on noisy analyzers and subjective judgments, making it difficult to measure whether security improves without sacrificing correctness. We address these gaps with CodeSecEval, an execution-based benchmark for secure code generation, comprising 255 Python tasks spanning 77 CWE categories. Each task provides paired insecure and secure implementations together with executable functional and vulnerability-targeted security tests, enabling precise and reproducible evaluation of secure code generation and insecure-code repair. Building on CodeSecEval, we propose SecAwareCoder, an agent-based framework that shifts code generation toward secure-by-construction synthesis. SecAwareCoder performs task-adaptive threat modeling to identify security-sensitive regions and derive task-grounded vulnerability hypotheses, uses these hypotheses to guide both constraint-aware code generation and security-aware test synthesis, and leverages execution feedback for targeted refinement. Experiments across multiple LLM backbones show that SecAwareCoder consistently improves Pass@1 and security robustness over prompting and analyzer-driven baselines, narrowing the security--correctness gap in LLM code generation.