# Enhancing Source Code Security with LLMs: Demystifying The Challenges and Generating Reliable Repairs

arXiv: https://arxiv.org/abs/2409.00571 · HTML: https://mlsanity.com/paper/2409.00571/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 44 of its 89 references that are in the map.
- First posted on arXiv 1 Sep 2024.

## Builds on (most cited first, 30 of 44)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Code Llama: Open Foundation Models for Code](https://mlsanity.com/paper/2308.12950/) (2023)
- [CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation](https://mlsanity.com/paper/2109.00859/) (2021)
- [The Falcon Series of Open Language Models](https://mlsanity.com/paper/2311.16867/) (2023)
- [CodeBLEU: a Method for Automatic Evaluation of Code Synthesis](https://mlsanity.com/paper/2009.10297/) (2020)
- [Asleep at the Keyboard? Assessing the Security of GitHub Copilot's Code Contributions](https://mlsanity.com/paper/2108.09293/) (2021)
- [Large Language Models for Software Engineering: A Systematic Literature Review](https://mlsanity.com/paper/2308.10620/) (2023)
- [CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learning](https://mlsanity.com/paper/2207.01780/) (2022)
- [CodeT5+: Open Code Large Language Models for Code Understanding and Generation](https://mlsanity.com/paper/2305.07922/) (2023)
- [Devign: Effective Vulnerability Identification by Learning Comprehensive Program Semantics via Graph Neural Networks](https://mlsanity.com/paper/1909.03496/) (2019)
- [Contextual Markov Decision Processes](https://mlsanity.com/paper/1502.02259/) (2015)
- [What Language Model Architecture and Pretraining Objective Work Best for Zero-Shot Generalization?](https://mlsanity.com/paper/2204.05832/) (2022)
- [MotionGPT: Finetuned LLMs Are General-Purpose Motion Generators](https://mlsanity.com/paper/2306.10900/) (2023)
- [Deep Learning based Vulnerability Detection: Are We There Yet?](https://mlsanity.com/paper/2009.07235/) (2020)
- [CodeGen2: Lessons for Training LLMs on Programming and Natural Languages](https://mlsanity.com/paper/2305.02309/) (2023)
- [Neural Machine Translation with Byte-Level Subwords](https://mlsanity.com/paper/1909.03341/) (2019)
- [Examining Zero-Shot Vulnerability Repair with Large Language Models](https://mlsanity.com/paper/2112.02125/) (2021)
- [Dos and Don'ts of Machine Learning in Computer Security](https://mlsanity.com/paper/2010.09470/) (2020)
- [LLMs Cannot Reliably Identify and Reason About Security Vulnerabilities (Yet?): A Comprehensive Evaluation, Framework, and Benchmarks](https://mlsanity.com/paper/2312.12575/) (2023)
- [The Programmer's Assistant: Conversational Interaction with a Large Language Model for Software Development](https://mlsanity.com/paper/2302.07080/) (2023)
- [Lost at C: A User Study on the Security Implications of Large Language Model Code Assistants](https://mlsanity.com/paper/2208.09727/) (2022)
- [VIGC: Visual Instruction Generation and Correction](https://mlsanity.com/paper/2308.12714/) (2023)
- [Repair Is Nearly Generation: Multilingual Program Repair with LLMs](https://mlsanity.com/paper/2208.11640/) (2022)
- [VulBERTa: Simplified Source Code Pre-Training for Vulnerability Detection](https://mlsanity.com/paper/2205.12424/) (2022)
- [Does BLEU Score Work for Code Migration?](https://mlsanity.com/paper/1906.04903/) (2019)
- [Large Language Models and Simple, Stupid Bugs](https://mlsanity.com/paper/2303.11455/) (2023)
- [A New Era in Software Security: Towards Self-Healing Software via Large Language Models and Formal Verification](https://mlsanity.com/paper/2305.14752/) (2023)

## Built on by (most cited first, 4 of 4)

- [Large Language Models for Code Generation: A Comprehensive Survey of Challenges, Techniques, Evaluation, and Applications](https://mlsanity.com/paper/2503.01245/) (2025)
- [Helping LLMs Improve Code Generation Using Feedback from Testing and Static Analysis](https://mlsanity.com/paper/2412.14841/) (2024)
- [CIPHER: Cryptographic Insecurity Profiling via Hybrid Evaluation of Responses](https://mlsanity.com/paper/2602.01438/) (2026)
- [SemOPT: Fixing Semantic Errors in LLM-based Optimization Modeling via Reward-Guided Search](https://mlsanity.com/paper/2609.37361/) (2026)

## Abstract (arXiv)

With the recent unprecedented advancements in Artificial Intelligence (AI) computing, progress in Large Language Models (LLMs) is accelerating rapidly, presenting challenges in establishing clear guidelines, particularly in the field of security. That being said, we thoroughly identify and describe three main technical challenges in the security and software engineering literature that spans the entire LLM workflow, namely; textbftextit(i) Data Collection and Labeling; textbftextit(ii) System Design and Learning; and textbftextit(iii) Performance Evaluation. Building upon these challenges, this paper introduces textttSecRepair, an instruction-based LLM system designed to reliably textitidentify, textitdescribe, and automatically textitrepair vulnerable source code. Our system is accompanied by a list of actionable guides on textbftextit(i) Data Preparation and Augmentation Techniques; textbftextit(ii) Selecting and Adapting state-of-the-art LLM Models; textbftextit(iii) Evaluation Procedures. textttSecRepair uses a reinforcement learning-based fine-tuning with a semantic reward that caters to the functionality and security aspects of the generated code. Our empirical analysis shows that textttSecRepair achieves a textit12% improvement in security code repair compared to other LLMs when trained using reinforcement learning. Furthermore, we demonstrate the capabilities of textttSecRepair in generating reliable, functional, and compilable security code repairs against real-world test cases using automated evaluation metrics.