# What Makes a Code Review Useful to OpenDev Developers? An Empirical Investigation

arXiv: https://arxiv.org/abs/2302.11686 · HTML: https://mlsanity.com/paper/2302.11686/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Feb 2023.

## Built on by (most cited first, 10 of 10)

- [CRScore: Grounding Automated Evaluation of Code Review Comments in Code Claims and Smells](https://mlsanity.com/paper/2409.19801/) (2024)
- [LAURA: Enhancing Code Review Generation with Context-Enriched Retrieval-Augmented LLM](https://mlsanity.com/paper/2512.01356/) (2025)
- [Leveraging Reviewer Experience in Code Review Comment Generation](https://mlsanity.com/paper/2409.10959/) (2024)
- [Are We All Using Agents the Same Way? An Empirical Study of Core and Peripheral Developers Use of Coding Agents](https://mlsanity.com/paper/2601.20106/) (2026)
- [Augmenting Large Language Models with Static Code Analysis for Automated Code Quality Improvements](https://mlsanity.com/paper/2506.10330/) (2025)
- [CodeWatcher: IDE Telemetry Data Extraction Tool for Understanding Coding Interactions with LLMs](https://mlsanity.com/paper/2510.11536/) (2025)
- [Rethinking Code Review in the Age of AI: A Vision for Agentic Code Review](https://mlsanity.com/paper/2605.17548/) (2026)
- [Is Agentic Code Review Helpful? Mining Developers' Feedback to CodeRabbit Reviews in the Wild](https://mlsanity.com/paper/2607.03316/) (2026)
- [Exploring the Potential of Large Language Models in Fine-Grained Review Comment Classification](https://mlsanity.com/paper/2508.09832/) (2025)
- [From Static to Dynamic: Benchmarking Real-World Code Review with MCR-Bench](https://mlsanity.com/paper/2608.27442/) (2026)

## Abstract (arXiv)

Context: Due to the association of significant efforts, even a minor improvement in the effectiveness of Code Reviews(CR) can incur significant savings for a software development organization. Aim: This study aims to develop a finer grain understanding of what makes a code review comment useful to OSS developers, to what extent a code review comment is considered useful to them, and how various contextual and participant-related factors influence its usefulness level. Method: On this goal, we have conducted a three-stage mixed-method study. We randomly selected 2,500 CR comments from the OpenDev Nova project and manually categorized the comments. We designed a survey of OpenDev developers to better understand their perspectives on useful CRs. Combining our survey-obtained scores with our manually labeled dataset, we trained two regression models - one to identify factors that influence the usefulness of CR comments and the other to identify factors that improve the odds of `Functional' defect identification over the others. Key findings: The results of our study suggest that a CR comment's usefulness is dictated not only by its technical contributions such as defect findings or quality improvement tips but also by its linguistic characteristics such as comprehensibility and politeness. While a reviewer's coding experience positively associates with CR usefulness, the number of mutual reviews, comment volume in a file, the total number of lines added /modified, and CR interval has the opposite associations. While authorship and reviewership experiences for the files under review have been the most popular attributes for reviewer recommendation systems, we do not find any significant association of those attributes with CR usefulness.