What Types of Code Review Comments Do Developers Most Frequently Resolve?
Not on the map. Why?
Abstract (arXiv)
Large language model (LLM)-powered code review automation tools have been introduced to generate code review comments. However, not all generated comments will drive code changes. Understanding what types of generated review comments are likely to trigger code changes is crucial for identifying those that are actionable. In this paper, we set out to investigate (1) the types of review comments written by humans and LLMs, and (2) the types of generated comments that are most frequently resolved by developers. To do so, we developed an LLM-as-a-Judge to automatically classify review comments based on our own taxonomy of five categories. Our empirical study confirms that (1) the LLM reviewer and human reviewers exhibit distinct strengths and weaknesses depending on the project context, and (2) readability, bugs, and maintainability-related comments had higher resolution rates than those focused on code design. These results suggest that a substantial proportion of LLM-generated comments are actionable and can be resolved by developers. Our work highlights the complementarity between LLM and human reviewers and offers suggestions to improve the practical effectiveness of LLM-powered code review tools.
Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). First posted on arXiv 6 Oct 2025.
Builds on
No reference list yet.
Built on by
Citing papers per year: 2025: 1; 2026: 6.
7 papers.
- 2026RovoDev Code Reviewer: A Large-Scale Online Evaluation of LLM-based Code Review Automation at Atlassiancited by 6
- 2026HalluJudge: A Reference-Free Hallucination Detection for Context Misalignment in Code Review Automationcited by 5
- 2025Issue-Oriented Agent-Based Framework for Automated Review Comment Generationcited by 4
- 2026Human-AI Synergy in Agentic Code Reviewcited by 3
- 2026Is Agentic Code Review Helpful? Mining Developers' Feedback to CodeRabbit Reviews in the Wildcited by 1
- 2026ConceptRM: The Quest to Mitigate Alert Fatigue through Consensus-Based Purity-Driven Data Cleaning for Reflection Modellingcited by 0
- 2026Code Broker: A Multi-Agent System for Automated Code Quality Assessmentcited by 0
- 2026Is Agentic Code Review Helpful? Mining Developers' Feedback to CodeRabbit Reviews in the Wild3 Jul 2026
- 2026Code Broker: A Multi-Agent System for Automated Code Quality Assessment25 Apr 2026
- 2026Human-AI Synergy in Agentic Code Review16 Mar 2026
- 2026ConceptRM: The Quest to Mitigate Alert Fatigue through Consensus-Based Purity-Driven Data Cleaning for Reflection Modelling9 Feb 2026
- 2026HalluJudge: A Reference-Free Hallucination Detection for Context Misalignment in Code Review Automation27 Jan 2026
- 2026RovoDev Code Reviewer: A Large-Scale Online Evaluation of LLM-based Code Review Automation at Atlassian3 Jan 2026
- 2025Issue-Oriented Agent-Based Framework for Automated Review Comment Generation1 Nov 2025