# Towards Practical Defect-Focused Automated Code Review

arXiv: https://arxiv.org/abs/2505.17928 · HTML: https://mlsanity.com/paper/2505.17928/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 58 references that are in the map.
- First posted on arXiv 23 May 2025.

## Builds on (most cited first, 27 of 27)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [Lost in the Middle: How Language Models Use Long Contexts](https://mlsanity.com/paper/2307.03172/) (2023)
- [RULER: What's the Real Context Size of Your Long-Context Language Models?](https://mlsanity.com/paper/2404.06654/) (2024)
- [AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration](https://mlsanity.com/paper/2306.00978/) (2023)
- [Large Language Model based Multi-Agents: A Survey of Progress and Challenges](https://mlsanity.com/paper/2402.01680/) (2024)
- [An Empirical Evaluation of Using Large Language Models for Automated Unit Test Generation](https://mlsanity.com/paper/2302.06527/) (2023)
- [Automating Code Review Activities by Large-Scale Pre-training](https://mlsanity.com/paper/2203.09095/) (2022)
- [ChatUniTest: A Framework for LLM-Based Test Generation](https://mlsanity.com/paper/2305.04764/) (2023)
- [SySeVR: A Framework for Using Deep Learning to Detect Software Vulnerabilities](https://mlsanity.com/paper/1807.06756/) (2018)
- [Using Pre-Trained Models to Boost Code Review Automation](https://mlsanity.com/paper/2201.06850/) (2022)
- [Towards Automating Code Review Activities](https://mlsanity.com/paper/2101.02518/) (2021)
- [LLaMA-Reviewer: Advancing Code Review Automation with Large Language Models through Parameter-Efficient Fine-Tuning](https://mlsanity.com/paper/2308.11148/) (2023)
- [LLMParser: An Exploratory Study on Using Large Language Models for Log Parsing](https://mlsanity.com/paper/2404.18001/) (2024)
- [GraphRAG under Fire](https://mlsanity.com/paper/2501.14050/) (2025)
- [AUGER: Automatically Generating Review Comments with Pre-training Models](https://mlsanity.com/paper/2208.08014/) (2022)
- [RobustKV: Defending Large Language Models against Jailbreak Attacks via KV Eviction](https://mlsanity.com/paper/2410.19937/) (2024)
- [CCT5: A Code-Change-Oriented Pre-Trained Model](https://mlsanity.com/paper/2305.10785/) (2023)
- [CORE: Automating Review Recommendation for Code Changes](https://mlsanity.com/paper/1912.09652/) (2019)
- [How to Refactor this Code? An Exploratory Study on Developer-ChatGPT Refactoring Conversations](https://mlsanity.com/paper/2402.06013/) (2024)
- [DeepCRCEval: Revisiting the Evaluation of Code Review Comment Generation](https://mlsanity.com/paper/2412.18291/) (2024)
- [KADEL: Knowledge-Aware Denoising Learning for Commit Message Generation](https://mlsanity.com/paper/2401.08376/) (2024)
- [Smart-LLaMA: Two-Stage Post-Training of Large Language Models for Smart Contract Vulnerability Detection and Explanation](https://mlsanity.com/paper/2411.06221/) (2024)
- [Improving the Learning of Code Review Successive Tasks with Cross-Task Knowledge Distillation](https://mlsanity.com/paper/2402.02063/) (2024)
- [Deep Learning-based Code Reviews: A Paradigm Shift or a Double-Edged Sword?](https://mlsanity.com/paper/2411.11401/) (2024)

## Built on by (most cited first, 3 of 3)

- [Intuition to Evidence: Measuring AI's True Impact on Developer Productivity](https://mlsanity.com/paper/2509.19708/) (2025)
- [Hallucinations in Code Change to Natural Language Generation: Prevalence and Evaluation of Detection Metrics](https://mlsanity.com/paper/2508.08661/) (2025)
- [Sphinx: Benchmarking and Modeling for LLM-Driven Pull Request Review](https://mlsanity.com/paper/2601.04252/) (2026)

## Abstract (arXiv)

The complexity of code reviews has driven efforts to automate review comments, but prior approaches oversimplify this task by treating it as snippet-level code-to-text generation and relying on text similarity metrics like BLEU for evaluation. These methods overlook repository context, real-world merge request evaluation, and defect detection, limiting their practicality. To address these issues, we explore the full automation pipeline within the online recommendation service of a company with nearly 400 million daily active users, analyzing industry-grade C++ codebases comprising hundreds of thousands of lines of code. We identify four key challenges: 1) capturing relevant context, 2) improving key bug inclusion (KBI), 3) reducing false alarm rates (FAR), and 4) integrating human workflows. To tackle these, we propose 1) code slicing algorithms for context extraction, 2) a multi-role LLM framework for KBI, 3) a filtering mechanism for FAR reduction, and 4) a novel prompt design for better human interaction. Our approach, validated on real-world merge requests from historical fault reports, achieves a 2x improvement over standard LLMs and a 10x gain over previous baselines. While the presented results focus on C++, the underlying framework design leverages language-agnostic principles (e.g., AST-based analysis), suggesting potential for broader applicability.