# From Static to Dynamic: Benchmarking Real-World Code Review with MCR-Bench

arXiv: https://arxiv.org/abs/2608.27442 · HTML: https://mlsanity.com/paper/2608.27442/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 34 of its 75 references that are in the map.
- First posted on arXiv 27 Aug 2026.

## Builds on (most cited first, 30 of 34)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [ChatDev: Communicative Agents for Software Development](https://mlsanity.com/paper/2307.07924/) (2023)
- [StarCoder 2 and The Stack v2: The Next Generation](https://mlsanity.com/paper/2402.19173/) (2024)
- [Large Language Models for Software Engineering: A Systematic Literature Review](https://mlsanity.com/paper/2308.10620/) (2023)
- [Automating Code Review Activities by Large-Scale Pre-training](https://mlsanity.com/paper/2203.09095/) (2022)
- [Towards an Understanding of Large Language Models in Software Engineering Tasks](https://mlsanity.com/paper/2308.11396/) (2023)
- [A Survey of Large Language Models for Code: Evolution, Benchmarking, and Future Trends](https://mlsanity.com/paper/2311.10372/) (2023)
- [Using Pre-Trained Models to Boost Code Review Automation](https://mlsanity.com/paper/2201.06850/) (2022)
- [Towards Automating Code Review Activities](https://mlsanity.com/paper/2101.02518/) (2021)
- [Can LLMs Replace Manual Annotation of Software Engineering Artifacts?](https://mlsanity.com/paper/2408.05534/) (2024)
- [CodeAgent: Autonomous Communicative Agents for Code Review](https://mlsanity.com/paper/2402.02172/) (2024)
- [HumanEvo: An Evolution-aware Benchmark for More Realistic Evaluation of Repository-level Code Generation](https://mlsanity.com/paper/2406.06918/) (2024)
- [BitsAI-CR: Automated Code Review via LLM in Practice](https://mlsanity.com/paper/2501.15134/) (2025)
- [Code Review Automation: Strengths and Weaknesses of the State of the Art](https://mlsanity.com/paper/2401.05136/) (2024)
- [Top General Performance = Top Domain Performance? DomainCodeBench: A Multi-domain Code Generation Benchmark](https://mlsanity.com/paper/2412.18573/) (2024)
- [What Makes a Code Review Useful to OpenDev Developers? An Empirical Investigation](https://mlsanity.com/paper/2302.11686/) (2023)
- [Towards an Understanding of Context Utilization in Code Intelligence](https://mlsanity.com/paper/2504.08734/) (2025)
- [Large Language Models are Qualified Benchmark Builders: Rebuilding Pre-Training Datasets for Advancing Code Intelligence Tasks](https://mlsanity.com/paper/2504.19444/) (2025)
- [Retrieval-Augmented Code Review Comment Generation](https://mlsanity.com/paper/2506.11591/) (2025)
- [SWR-Bench: Assessing LLM Performance in Real-World Code Review Comment Generation](https://mlsanity.com/paper/2509.01494/) (2025)
- [Toward Effective Secure Code Reviews: An Empirical Study of Security-Related Coding Weaknesses](https://mlsanity.com/paper/2311.16396/) (2023)
- [CodeFuse-CR-Bench: A Comprehensiveness-aware Benchmark for End-to-End Code Review Evaluation in Python Projects](https://mlsanity.com/paper/2509.14856/) (2025)
- [iCodeReviewer: Improving Secure Code Review with Mixture of Prompts](https://mlsanity.com/paper/2510.12186/) (2025)
- [SecureReviewer: Enhancing Large Language Models for Secure Code Review through Secure-aware Fine-tuning](https://mlsanity.com/paper/2510.26457/) (2025)
- [AACR-Bench: Evaluating Automatic Code Review with Holistic Repository-Level Context](https://mlsanity.com/paper/2601.19494/) (2026)
- [RepoReasoner: Evaluating Repository-Level Code Reasoning Ability of Long-Context Language Models](https://mlsanity.com/paper/2607.25996/) (2026)
- [Understanding Practitioners' Expectations on Clear Code Review Comments](https://mlsanity.com/paper/2410.06515/) (2024)
- [Deep Assessment of Code Review Generation Approaches: Beyond Lexical Similarity](https://mlsanity.com/paper/2501.05176/) (2025)
- [Issue-Oriented Agent-Based Framework for Automated Review Comment Generation](https://mlsanity.com/paper/2511.00517/) (2025)

## Abstract (arXiv)

In real-world software development, code review typically involves iterative interactions between developers and reviewers to improve software quality, making the process costly and time-consuming. Although recent work explores large language models (LLMs) for automated code review, most approaches oversimplify code review into a single-round, static decision task, which fails to capture the multi-round interactive nature and the complex problem-solving processes inherent in realistic review scenarios. To bridge this gap, we introduce MCR-Bench, the first defect state-aware benchmark designed for realistic multi-round code review. MCR-Bench covers five commonly-used programming languages and consists of 2,269 real-world multi-round code review tasks, each of which is annotated with fine-grained defect information and cross-round state labels. Each task in MCR-Bench is equipped with fine-grained defect metadata (e.g., description, type, severity) alongside dynamic state annotations, capturing the complete evolutionary trajectory of a defect throughout the multi-round process. We obtain several findings through extensive experiments on MCR-Bench with mainstream LLMs. (1) Limited overall capability: experiments reveal that mainstream LLMs exhibit limited overall performance in defect detection and defect lifecycle state tracking, with performance degrading significantly as the number of interaction rounds increases; (2) Defect-sensitive performance: LLMs' performance varies substantially across different defect types and severity levels, with semantically complex or low-salience defects being significantly more likely to be missed; (3) Underlying Failure Mechanisms: our in-depth error analysis dissects the distinct drivers of false positives and false negatives, revealing critical weaknesses such as cross-round temporal misalignment and inadequate long-range memory.