# Previously on... Automating Code Review

arXiv: https://arxiv.org/abs/2508.18003 · HTML: https://mlsanity.com/paper/2508.18003/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 73 references that are in the map.
- First posted on arXiv 25 Aug 2025.

## Builds on (most cited first, 28 of 28)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [A Call for Clarity in Reporting BLEU Scores](https://mlsanity.com/paper/1804.08771/) (2018)
- [CodeBERT: A Pre-Trained Model for Programming and Natural Languages](https://mlsanity.com/paper/2002.08155/) (2020)
- [CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation](https://mlsanity.com/paper/2109.00859/) (2021)
- [EDA: Easy Data Augmentation Techniques for Boosting Performance on Text Classification Tasks](https://mlsanity.com/paper/1901.11196/) (2019)
- [CodeSearchNet Challenge: Evaluating the State of Semantic Code Search](https://mlsanity.com/paper/1909.09436/) (2019)
- [CodeBLEU: a Method for Automatic Evaluation of Code Synthesis](https://mlsanity.com/paper/2009.10297/) (2020)
- [A Survey of Machine Learning for Big Code and Naturalness](https://mlsanity.com/paper/1709.06182/) (2017)
- [Automating Code Review Activities by Large-Scale Pre-training](https://mlsanity.com/paper/2203.09095/) (2022)
- [Using Pre-Trained Models to Boost Code Review Automation](https://mlsanity.com/paper/2201.06850/) (2022)
- [Big Code != Big Vocabulary: Open-Vocabulary Models for Source Code](https://mlsanity.com/paper/2003.07914/) (2020)
- [Towards Automating Code Review Activities](https://mlsanity.com/paper/2101.02518/) (2021)
- [On Learning Meaningful Code Changes via Neural Machine Translation](https://mlsanity.com/paper/1901.09102/) (2019)
- [LLaMA-Reviewer: Advancing Code Review Automation with Large Language Models through Parameter-Efficient Fine-Tuning](https://mlsanity.com/paper/2308.11148/) (2023)
- [CodeTrans: Towards Cracking the Language of Silicon's Code Through Self-Supervised Deep Learning and High Performance Computing](https://mlsanity.com/paper/2104.02443/) (2021)
- [OpenNMT: Neural Machine Translation Toolkit](https://mlsanity.com/paper/1805.11462/) (2018)
- [AUGER: Automatically Generating Review Comments with Pre-training Models](https://mlsanity.com/paper/2208.08014/) (2022)
- [Code Review Automation: Strengths and Weaknesses of the State of the Art](https://mlsanity.com/paper/2401.05136/) (2024)
- [Oreo: Detection of Clones in the Twilight Zone](https://mlsanity.com/paper/1806.05837/) (2018)
- [CORE: Automating Review Recommendation for Code Changes](https://mlsanity.com/paper/1912.09652/) (2019)
- [On the Importance of Strong Baselines in Bayesian Deep Learning](https://mlsanity.com/paper/1811.09385/) (2018)
- [Generation-based Code Review Automation: How Far Are We?](https://mlsanity.com/paper/2303.07221/) (2023)
- [Nudge: Accelerating Overdue Pull Requests Towards Completion](https://mlsanity.com/paper/2011.12468/) (2020)
- [A Roadmap on Modern Code Review: Challenges and Opportunities](https://mlsanity.com/paper/2405.18216/) (2024)
- [First Come First Served: The Impact of File Position on Code Review](https://mlsanity.com/paper/2208.04259/) (2022)
- [Turn Tree into Graph: Automatic Code Review via Simplified AST Driven Graph Convolutional Network](https://mlsanity.com/paper/2202.07946/) (2022)

## Built on by (most cited first, 1 of 1)

- [Enhancing Large Language Models with Retrieval Augmented Generation for Software Testing and Inspection Automation](https://mlsanity.com/paper/2604.15270/) (2026)

## Abstract (arXiv)

Modern Code Review (MCR) is a standard practice in software engineering, yet it demands substantial time and resource investments. Recent research has increasingly explored automating core review tasks using machine learning (ML) and deep learning (DL). As a result, there is substantial variability in task definitions, datasets, and evaluation procedures. This study provides the first comprehensive analysis of MCR automation research, aiming to characterize the field's evolution, formalize learning tasks, highlight methodological challenges, and offer actionable recommendations to guide future research. Focusing on the primary code review tasks, we systematically surveyed 691 publications and identified 24 relevant studies published between May 2015 and April 2024. Each study was analyzed in terms of tasks, models, metrics, baselines, results, validity concerns, and artifact availability. In particular, our analysis reveals significant potential for standardization, including 48 task metric combinations, 22 of which were unique to their original paper, and limited dataset reuse. We highlight challenges and derive concrete recommendations for examples such as the temporal bias threat, which are rarely addressed so far. Our work contributes to a clearer overview of the field, supports the framing of new research, helps to avoid pitfalls, and promotes greater standardization in evaluation practices.