# Improving Automated Secure Code Reviews: A Synthetic Dataset for Code Vulnerability Flaws

arXiv: https://arxiv.org/abs/2504.16310 · HTML: https://mlsanity.com/paper/2504.16310/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Code and instruction data synthesis, one of 13 directions in Synthetic data and self-improvement for LLMs.
- Builds on 19 of its 26 references that are in the map.
- First posted on arXiv 22 Apr 2025.

## Placed in

- Synthetic data and self-improvement for LLMs / [Code and instruction data synthesis](https://mlsanity.com/synth/code-and-instruction-data-synthesis/)

## Builds on (most cited first, 19 of 19)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Unnatural Instructions: Tuning Language Models with (Almost) No Human Labor](https://mlsanity.com/paper/2212.09689/) (2022)
- [Large Language Model as Attributed Training Data Generator: A Tale of Diversity and Bias](https://mlsanity.com/paper/2306.15895/) (2023)
- [ZeroGen: Efficient Zero-shot Learning via Dataset Generation](https://mlsanity.com/paper/2202.07922/) (2022)
- [Automating Code Review Activities by Large-Scale Pre-training](https://mlsanity.com/paper/2203.09095/) (2022)
- [Using Pre-Trained Models to Boost Code Review Automation](https://mlsanity.com/paper/2201.06850/) (2022)
- [Towards Automating Code Review Activities](https://mlsanity.com/paper/2101.02518/) (2021)
- [LLaMA-Reviewer: Advancing Code Review Automation with Large Language Models through Parameter-Efficient Fine-Tuning](https://mlsanity.com/paper/2308.11148/) (2023)
- [Exploring the Potential of ChatGPT in Automated Code Refinement: An Empirical Study](https://mlsanity.com/paper/2309.08221/) (2023)
- [Synthetic Dialogue Dataset Generation using LLM Agents](https://mlsanity.com/paper/2401.17461/) (2024)
- [AUGER: Automatically Generating Review Comments with Pre-training Models](https://mlsanity.com/paper/2208.08014/) (2022)
- [Code Review Automation: Strengths and Weaknesses of the State of the Art](https://mlsanity.com/paper/2401.05136/) (2024)
- [Improving Automated Code Reviews: Learning from Experience](https://mlsanity.com/paper/2402.03777/) (2024)
- [CodeEditor: Learning to Edit Source Code with Pre-trained Models](https://mlsanity.com/paper/2210.17040/) (2022)
- [Why Security Defects Go Unnoticed during Code Reviews? A Case-Control Study of the Chromium OS Project](https://mlsanity.com/paper/2102.06909/) (2021)
- [Mining Code Review Data to Understand Waiting Times Between Acceptance and Merging: An Empirical Analysis](https://mlsanity.com/paper/2203.05048/) (2022)
- [Security Defect Detection via Code Review: A Study of the OpenStack and Qt Communities](https://mlsanity.com/paper/2307.02326/) (2023)
- [Software Security during Modern Code Review: The Developer's Perspective](https://mlsanity.com/paper/2208.04261/) (2022)

## Abstract (arXiv)

Automation of code reviews using AI models has garnered substantial attention in the software engineering community as a strategy to reduce the cost and effort associated with traditional peer review processes. These models are typically trained on extensive datasets of real-world code reviews that address diverse software development concerns, including testing, refactoring, bug fixes, performance optimization, and maintainability improvements. However, a notable limitation of these datasets is the under representation of code vulnerabilities, critical flaws that pose significant security risks, with security-focused reviews comprising a small fraction of the data. This scarcity of vulnerability-specific data restricts the effectiveness of AI models in identifying and commenting on security-critical code. To address this issue, we propose the creation of a synthetic dataset consisting of vulnerability-focused reviews that specifically comment on security flaws. Our approach leverages Large Language Models (LLMs) to generate human-like code review comments for vulnerabilities, using insights derived from code differences and commit messages. To evaluate the usefulness of the generated synthetic dataset, we plan to use it to fine-tune three existing code review models. We anticipate that the synthetic dataset will improve the performance of the original code review models.