# Code Review Automation: Strengths and Weaknesses of the State of the Art

arXiv: https://arxiv.org/abs/2401.05136 · HTML: https://mlsanity.com/paper/2401.05136/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Jan 2024.

## Built on by (most cited first, 20 of 20)

- [Automatic Programming: Large Language Models and Beyond](https://mlsanity.com/paper/2405.02213/) (2024)
- [AI-Assisted Assessment of Coding Practices in Modern Code Review](https://mlsanity.com/paper/2405.13565/) (2024)
- [LAURA: Enhancing Code Review Generation with Context-Enriched Retrieval-Augmented LLM](https://mlsanity.com/paper/2512.01356/) (2025)
- [Combining Large Language Models with Static Analyzers for Code Review Generation](https://mlsanity.com/paper/2502.06633/) (2025)
- [RovoDev Code Reviewer: A Large-Scale Online Evaluation of LLM-based Code Review Automation at Atlassian](https://mlsanity.com/paper/2601.01129/) (2026)
- [Leveraging Reviewer Experience in Code Review Comment Generation](https://mlsanity.com/paper/2409.10959/) (2024)
- [Issue-Oriented Agent-Based Framework for Automated Review Comment Generation](https://mlsanity.com/paper/2511.00517/) (2025)
- [Towards Goal-oriented Prompt Engineering for Large Language Models: A Survey](https://mlsanity.com/paper/2401.14043/) (2024)
- [CodeReviewQA: The Code Review Comprehension Assessment for Large Language Models](https://mlsanity.com/paper/2503.16167/) (2025)
- [DeputyDev -- AI Powered Developer Assistant: Breaking the Code Review Logjam through Contextual AI to Boost Developer Productivity](https://mlsanity.com/paper/2508.09676/) (2025)
- [Human-AI Synergy in Agentic Code Review](https://mlsanity.com/paper/2603.15911/) (2026)
- [SWE-PRBench: Benchmarking AI Code Review Quality Against Pull Request Feedback](https://mlsanity.com/paper/2603.26130/) (2026)
- [Previously on... Automating Code Review](https://mlsanity.com/paper/2508.18003/) (2025)
- [Code Review Without Borders: Evaluating Synthetic vs. Real Data for Review Recommendation](https://mlsanity.com/paper/2509.04810/) (2025)
- [Improving Automated Secure Code Reviews: A Synthetic Dataset for Code Vulnerability Flaws](https://mlsanity.com/paper/2504.16310/) (2025)
- [On Assessing the Relevance of Code Reviews Authored by Generative Models](https://mlsanity.com/paper/2512.15466/) (2025)
- [Fine-grained Approaches for Confidence Calibration of LLMs in Automated Code Revision](https://mlsanity.com/paper/2604.06723/) (2026)
- [Archer: Towards Agentic Review for Compiler Optimizations](https://mlsanity.com/paper/2607.01808/) (2026)
- [Evaluating the Impact of Explainable AI on Trust in AI-Assisted Code Review](https://mlsanity.com/paper/2607.24601/) (2026)
- [From Static to Dynamic: Benchmarking Real-World Code Review with MCR-Bench](https://mlsanity.com/paper/2608.27442/) (2026)

## Abstract (arXiv)

The automation of code review has been tackled by several researchers with the goal of reducing its cost. The adoption of deep learning in software engineering pushed the automation to new boundaries, with techniques imitating developers in generative tasks, such as commenting on a code change as a reviewer would do or addressing a reviewer's comment by modifying code. The performance of these techniques is usually assessed through quantitative metrics, e.g., the percentage of instances in the test set for which correct predictions are generated, leaving many open questions on the techniques' capabilities. For example, knowing that an approach is able to correctly address a reviewer's comment in 10% of cases is of little value without knowing what was asked by the reviewer: What if in all successful cases the code change required to address the comment was just the removal of an empty line? In this paper we aim at characterizing the cases in which three code review automation techniques tend to succeed or fail in the two above-described tasks. The study has a strong qualitative focus, with ~105 man-hours of manual inspection invested in manually analyzing correct and wrong predictions generated by the three techniques, for a total of 2,291 inspected predictions. The output of this analysis are two taxonomies reporting, for each of the two tasks, the types of code changes on which the experimented techniques tend to succeed or to fail, pointing to areas for future work. A result of our manual analysis was also the identification of several issues in the datasets used to train and test the experimented techniques. Finally, we assess the importance of researching in techniques specialized for code review automation by comparing their performance with ChatGPT, a general purpose large language model, finding that ChatGPT struggles in commenting code as a human reviewer would do.