# CR-Bench: Evaluating the Real-World Utility of AI Code Review Agents

arXiv: https://arxiv.org/abs/2603.11078 · HTML: https://mlsanity.com/paper/2603.11078/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use.
- Builds on 9 of its 30 references that are in the map.
- First posted on arXiv 10 Mar 2026.

## Placed in

- LLM agents and tool use / [Software engineering coding agents](https://mlsanity.com/agents/software-engineering-coding-agents/)

## Builds on (most cited first, 9 of 9)

- [Reflexion: Language Agents with Verbal Reinforcement Learning](https://mlsanity.com/paper/2303.11366/) (2023)
- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment](https://mlsanity.com/paper/2303.16634/) (2023)
- [A Comprehensive Study on Deep Learning Bug Characteristics](https://mlsanity.com/paper/1906.01388/) (2019)
- [Using Pre-Trained Models to Boost Code Review Automation](https://mlsanity.com/paper/2201.06850/) (2022)
- [Towards Automating Code Review Activities](https://mlsanity.com/paper/2101.02518/) (2021)
- [CodeAgent: Autonomous Communicative Agents for Code Review](https://mlsanity.com/paper/2402.02172/) (2024)
- [SWR-Bench: Assessing LLM Performance in Real-World Code Review Comment Generation](https://mlsanity.com/paper/2509.01494/) (2025)
- [Harnessing Large Language Models for Curated Code Reviews](https://mlsanity.com/paper/2502.03425/) (2025)

## Built on by (most cited first, 3 of 3)

- [SWE-Review: Closing the Loop on Issue Resolution with Agentic Code Review](https://mlsanity.com/paper/2607.06065/) (2026)
- [Code Review Agent Benchmark](https://mlsanity.com/paper/2603.23448/) (2026)
- [CRJudgeBench: Can AI Detect Plausible but Invalid Code Reviews?](https://mlsanity.com/paper/2609.37216/) (2026)

## Abstract (arXiv)

Recent advances in frontier large language models have enabled code review agents that operate in open-ended, reasoning-intensive settings. However, the lack of standardized benchmarks and granular evaluation protocols makes it difficult to assess behavior of code review agents beyond coarse success metrics, particularly for tasks where false positives are costly. To address this gap, we introduce CR-Bench, a benchmarking dataset, and CR-Evaluator, a fine-grained evaluation pipeline for code review agents. Using these tools, we conduct a preliminary study evaluating both a single-shot agent and a Reflexion-based agent across two frontier models. We find that code review agents can exhibit a low signal-to-noise ratio when designed to identify all hidden issues, obscuring true progress and developer productivity when measured solely by resolution rates. Our analysis identifies the hidden trade-off between issue resolution and spurious findings, revealing a frontier that constrains effective agent design. Together, CR-Bench and CR-Evaluator provide a timely foundation for studying and developing code review agents as LLM-based systems transition from controlled benchmarks to real-world software engineering workflows.