mlsanityData to 7 Oct 2026

SoK: Understanding (New) Security Issues Across AI4Code Use Cases

arXiv 2512.18456 ↗First posted 20 Dec 2025cs.CR cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

AI-for-Code (AI4Code) systems are reshaping software engineering, with tools like GitHub Copilot accelerating code generation, translation, and vulnerability detection. Alongside these advances, however, security risks remain pervasive: insecure outputs, biased benchmarks, and susceptibility to adversarial manipulation undermine their reliability. This SoK surveys the landscape of AI4Code security across three core applications, identifying recurring gaps: benchmark dominance by Python and toy problems, lack of standardized security datasets, data leakage in evaluation, and fragile adversarial robustness. A comparative study of six state-of-the-art models illustrates these challenges: insecure patterns persist in code generation, vulnerability detection is brittle to semantic-preserving attacks, fine-tuning often misaligns security objectives, and code translation yields uneven security benefits. From this analysis, we distill three forward paths: embedding secure-by-default practices in code generation, building robust and comprehensive detection benchmarks, and leveraging translation as a route to security-enhanced languages. We call for a shift toward security-first AI4Code, where vulnerability mitigation and robustness are embedded throughout the development life cycle.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 95 of its 142 references that are in the map. First posted on arXiv 20 Dec 2025.

Builds on

95 of 142 references.

References · 95

  1. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  2. 2021Program Synthesis with Large Language Modelscited by 3,509
  3. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 1,906
  4. 2022Competition-Level Code Generation with AlphaCodecited by 1,427
  5. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,332
  6. 2020CodeBERT: A Pre-Trained Model for Programming and Natural Languagescited by 1,286
  7. 2018Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Taskcited by 1,120
  8. 2023Teaching Large Language Models to Self-Debugcited by 963
  9. 2021CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generationcited by 890
  10. 2021Measuring Coding Challenge Competence With APPScited by 879
  11. 2022CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesiscited by 876
  12. 2023StarCoder: may the source be with you!cited by 790
38 more
  1. 2023Large Language Modelscited by 774
  2. 2021CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generationcited by 732
  3. 2023WizardCoder: Empowering Code Large Language Models with Evol-Instructcited by 728
  4. 2019CodeSearchNet Challenge: Evaluating the State of Semantic Code Searchcited by 638
  5. 2020GraphCodeBERT: Pre-training Code Representations with Data Flowcited by 600
  6. 2024BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructionscited by 527
  7. 2020CodeBLEU: a Method for Automatic Evaluation of Code Synthesiscited by 468
  8. 2022DS-1000: A Natural and Reliable Benchmark for Data Science Code Generationcited by 442
  9. 2021Asleep at the Keyboard? Assessing the Security of GitHub Copilot's Code Contributionscited by 417
  10. 2023Large Language Models for Software Engineering: A Systematic Literature Reviewcited by 389
  11. 2021Unified Pre-training for Program Understanding and Generationcited by 361
  12. 2022UniXcoder: Unified Cross-Modal Pre-training for Code Representationcited by 355
  13. 2022CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learningcited by 342
  14. 2023CodeT5+: Open Code Large Language Models for Code Understanding and Generationcited by 327
  15. 2024CodeAgent: Enhancing Code Generation with Tool-Integrated Agent Systems for Real-World Repo-level Coding Challengescited by 278
  16. 2020Unsupervised Translation of Programming Languagescited by 262
  17. 2021CodeNet: A Large-Scale AI for Code Dataset for Learning a Diversity of Coding Taskscited by 199
  18. 2022Do Users Write More Insecure Code with AI Assistants?cited by 186
  19. 2023CrossCodeEval: A Diverse and Multilingual Benchmark for Cross-File Code Completioncited by 181
  20. 2018Learning to Mine Aligned Code and Natural Language Pairs from Stack Overflowcited by 165
  21. 2022Synchromesh: Reliable code generation from pre-trained language modelscited by 156
  22. 2023Large Language Models for Code: Security Hardening and Adversarial Testingcited by 149
  23. 2020Deep Learning based Vulnerability Detection: Are We There Yet?cited by 142
  24. 2024Vulnerability Detection with Code Language Models: How Far Are We?cited by 133
  25. 2023LongCoder: A Long-Range Pre-trained Language Model for Code Completioncited by 110
  26. 2023DiverseVul: A New Vulnerable Source Code Dataset for Deep Learning Based Vulnerability Detectioncited by 98
  27. 2023LLMs Cannot Reliably Identify and Reason About Security Vulnerabilities (Yet?): A Comprehensive Evaluation, Framework, and Benchmarkscited by 94
  28. 2018Tree-to-tree Neural Networks for Program Translationcited by 93
  29. 2021CVEfixes: Automated Collection of Vulnerabilities and Their Fixes from Open-Source Softwarecited by 89
  30. 2022Natural Language to Code Generation in Interactive Data Science Notebookscited by 86
  31. 2021Leveraging Automated Unit Tests for Unsupervised Code Translationcited by 82
  32. 2024DebugBench: Evaluating Debugging Capability of Large Language Modelscited by 74
  33. 2024RepoAgent: An LLM-Powered Open-Source Framework for Repository-level Code Documentation Generationcited by 69
  34. 2018DeepBugs: A Learning Approach to Name-based Bug Detectioncited by 65
  35. 2021AVATAR: A Parallel Corpus for Java-Python Program Translationcited by 59
  36. 2023Understanding the Effectiveness of Large Language Models in Detecting Security Vulnerabilitiescited by 58
  37. 2024LLM4Vuln: A Unified Evaluation Framework for Decoupling and Enhancing LLMs' Vulnerability Reasoningcited by 57
  38. 2024Exploring and Unleashing the Power of Large Language Models in Automated Code Translationcited by 57

50 most cited of 95 All 95 (JSON)

Built on by

Not cited yet.