# Does AI Code Review Lead to Code Changes? A Case Study of GitHub Actions

arXiv: https://arxiv.org/abs/2508.18771 · HTML: https://mlsanity.com/paper/2508.18771/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 Aug 2025.

## Built on by (most cited first, 7 of 7)

- [Investigating Autonomous Agent Contributions in the Wild: Activity Patterns and Code Change over Time](https://mlsanity.com/paper/2604.00917/) (2026)
- [Human-AI Synergy in Agentic Code Review](https://mlsanity.com/paper/2603.15911/) (2026)
- [On the Footprints of Reviewer Bots Feedback on Agentic Pull Requests in OSS GitHub Repositories](https://mlsanity.com/paper/2604.24450/) (2026)
- [When AI Reviews Its Own Code: Recursive Self-Training Collapse in Code LLMs](https://mlsanity.com/paper/2606.28438/) (2026)
- [Enhancing Large Language Models with Retrieval Augmented Generation for Software Testing and Inspection Automation](https://mlsanity.com/paper/2604.15270/) (2026)
- [From Human-Centric to Agentic Code Review: The Impact of Different Generations of Generative AI Technology on Review Quality](https://mlsanity.com/paper/2607.13196/) (2026)
- [Specifying and Maintaining Agentic Workflows: An Empirical Study of GitHub Agentic Workflows](https://mlsanity.com/paper/2609.27263/) (2026)

## Abstract (arXiv)

AI-based code review tools automatically review and comment on pull requests to improve code quality. Despite their growing presence, little is known about their actual impact. We present a large-scale empirical study of 16 popular AI-based code review actions for GitHub workflows, analyzing more than 22,000 review comments in 178 repositories. We investigate (1) how these tools are adopted and configured, (2) whether their comments lead to code changes, and (3) which factors influence their effectiveness. We develop a two-stage LLM-assisted framework to determine whether review comments are addressed, and use interpretable machine learning to identify influencing factors. Our findings show that, while adoption is growing, effectiveness varies widely. Comments that are concise, contain code snippets, and are manually triggered, particularly those from hunk-level review tools, are more likely to result in code changes. These results highlight the importance of careful tool design and suggest directions for improving AI-based code review systems.