SoK: Understanding (New) Security Issues Across AI4Code Use Cases
Not on the map. Why?
Abstract (arXiv)
AI-for-Code (AI4Code) systems are reshaping software engineering, with tools like GitHub Copilot accelerating code generation, translation, and vulnerability detection. Alongside these advances, however, security risks remain pervasive: insecure outputs, biased benchmarks, and susceptibility to adversarial manipulation undermine their reliability. This SoK surveys the landscape of AI4Code security across three core applications, identifying recurring gaps: benchmark dominance by Python and toy problems, lack of standardized security datasets, data leakage in evaluation, and fragile adversarial robustness. A comparative study of six state-of-the-art models illustrates these challenges: insecure patterns persist in code generation, vulnerability detection is brittle to semantic-preserving attacks, fine-tuning often misaligns security objectives, and code translation yields uneven security benefits. From this analysis, we distill three forward paths: embedding secure-by-default practices in code generation, building robust and comprehensive detection benchmarks, and leveraging translation as a route to security-enhanced languages. We call for a shift toward security-first AI4Code, where vulnerability mitigation and robustness are embedded throughout the development life cycle.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 95 of its 142 references that are in the map. First posted on arXiv 20 Dec 2025.
Builds on
95 of 142 references.
References · 95
- 2021Evaluating Large Language Models Trained on Codecited by 8,053
- 2021Program Synthesis with Large Language Modelscited by 3,509
- 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 1,906
- 2022Competition-Level Code Generation with AlphaCodecited by 1,427
- 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,332
- 2020CodeBERT: A Pre-Trained Model for Programming and Natural Languagescited by 1,286
- 2018Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Taskcited by 1,120
- 2023Teaching Large Language Models to Self-Debugcited by 963
- 2021CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generationcited by 890
- 2021Measuring Coding Challenge Competence With APPScited by 879
- 2022CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesiscited by 876
- 2023StarCoder: may the source be with you!cited by 790
38 more
- 2023Large Language Modelscited by 774
- 2021CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generationcited by 732
- 2023WizardCoder: Empowering Code Large Language Models with Evol-Instructcited by 728
- 2019CodeSearchNet Challenge: Evaluating the State of Semantic Code Searchcited by 638
- 2020GraphCodeBERT: Pre-training Code Representations with Data Flowcited by 600
- 2024BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructionscited by 527
- 2020CodeBLEU: a Method for Automatic Evaluation of Code Synthesiscited by 468
- 2022DS-1000: A Natural and Reliable Benchmark for Data Science Code Generationcited by 442
- 2021Asleep at the Keyboard? Assessing the Security of GitHub Copilot's Code Contributionscited by 417
- 2023Large Language Models for Software Engineering: A Systematic Literature Reviewcited by 389
- 2021Unified Pre-training for Program Understanding and Generationcited by 361
- 2022UniXcoder: Unified Cross-Modal Pre-training for Code Representationcited by 355
- 2022CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learningcited by 342
- 2023CodeT5+: Open Code Large Language Models for Code Understanding and Generationcited by 327
- 2024CodeAgent: Enhancing Code Generation with Tool-Integrated Agent Systems for Real-World Repo-level Coding Challengescited by 278
- 2020Unsupervised Translation of Programming Languagescited by 262
- 2021CodeNet: A Large-Scale AI for Code Dataset for Learning a Diversity of Coding Taskscited by 199
- 2022Do Users Write More Insecure Code with AI Assistants?cited by 186
- 2023CrossCodeEval: A Diverse and Multilingual Benchmark for Cross-File Code Completioncited by 181
- 2018Learning to Mine Aligned Code and Natural Language Pairs from Stack Overflowcited by 165
- 2022Synchromesh: Reliable code generation from pre-trained language modelscited by 156
- 2023Large Language Models for Code: Security Hardening and Adversarial Testingcited by 149
- 2020Deep Learning based Vulnerability Detection: Are We There Yet?cited by 142
- 2024Vulnerability Detection with Code Language Models: How Far Are We?cited by 133
- 2023LongCoder: A Long-Range Pre-trained Language Model for Code Completioncited by 110
- 2023DiverseVul: A New Vulnerable Source Code Dataset for Deep Learning Based Vulnerability Detectioncited by 98
- 2023LLMs Cannot Reliably Identify and Reason About Security Vulnerabilities (Yet?): A Comprehensive Evaluation, Framework, and Benchmarkscited by 94
- 2018Tree-to-tree Neural Networks for Program Translationcited by 93
- 2021CVEfixes: Automated Collection of Vulnerabilities and Their Fixes from Open-Source Softwarecited by 89
- 2022Natural Language to Code Generation in Interactive Data Science Notebookscited by 86
- 2021Leveraging Automated Unit Tests for Unsupervised Code Translationcited by 82
- 2024DebugBench: Evaluating Debugging Capability of Large Language Modelscited by 74
- 2024RepoAgent: An LLM-Powered Open-Source Framework for Repository-level Code Documentation Generationcited by 69
- 2018DeepBugs: A Learning Approach to Name-based Bug Detectioncited by 65
- 2021AVATAR: A Parallel Corpus for Java-Python Program Translationcited by 59
- 2023Understanding the Effectiveness of Large Language Models in Detecting Security Vulnerabilitiescited by 58
- 2024LLM4Vuln: A Unified Evaluation Framework for Decoupling and Enhancing LLMs' Vulnerability Reasoningcited by 57
- 2024Exploring and Unleashing the Power of Large Language Models in Automated Code Translationcited by 57
50 most cited of 95 All 95 (JSON)
Built on by
Not cited yet.