Enhancing Source Code Security with LLMs: Demystifying The Challenges and Generating Reliable Repairs

With the recent unprecedented advancements in Artificial Intelligence (AI) computing, progress in Large Language Models (LLMs) is accelerating rapidly, presenting challenges in establishing clear guidelines, particularly in the field of security. That being said, we thoroughly identify and describe three main technical challenges in the security and software engineering literature that spans the entire LLM workflow, namely; textbftextit(i) Data Collection and Labeling; textbftextit(ii) System Design and Learning; and textbftextit(iii) Performance Evaluation. Building upon these challenges, this paper introduces textttSecRepair, an instruction-based LLM system designed to reliably textitidentify, textitdescribe, and automatically textitrepair vulnerable source code. Our system is accompanied by a list of actionable guides on textbftextit(i) Data Preparation and Augmentation Techniques; textbftextit(ii) Selecting and Adapting state-of-the-art LLM Models; textbftextit(iii) Evaluation Procedures. textttSecRepair uses a reinforcement learning-based fine-tuning with a semantic reward that caters to the functionality and security aspects of the generated code. Our empirical analysis shows that textttSecRepair achieves a textit12% improvement in security code repair compared to other LLMs when trained using reinforcement learning. Furthermore, we demonstrate the capabilities of textttSecRepair in generating reliable, functional, and compilable security code repairs against real-world test cases using automated evaluation metrics.