# SWR-Bench: Assessing LLM Performance in Real-World Code Review Comment Generation

arXiv: https://arxiv.org/abs/2509.01494 · HTML: https://mlsanity.com/paper/2509.01494/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Sep 2025.

## Built on by (most cited first, 6 of 6)

- [AACR-Bench: Evaluating Automatic Code Review with Holistic Repository-Level Context](https://mlsanity.com/paper/2601.19494/) (2026)
- [Code Review Agent Benchmark](https://mlsanity.com/paper/2603.23448/) (2026)
- [CR-Bench: Evaluating the Real-World Utility of AI Code Review Agents](https://mlsanity.com/paper/2603.11078/) (2026)
- [MathlibPR: Pull Request Merge-Readiness Benchmark for Formal Mathematical Libraries](https://mlsanity.com/paper/2605.07147/) (2026)
- [From Static to Dynamic: Benchmarking Real-World Code Review with MCR-Bench](https://mlsanity.com/paper/2608.27442/) (2026)
- [SWE-DeceptionBench: Benchmarking Deceptive Behaviors of LLMs Across the Software Development Lifecycle](https://mlsanity.com/paper/2610.13626/) (2026)

## Abstract (arXiv)

Automated Code Review (ACR) is crucial for software quality, yet existing benchmarks often fail to reflect real-world complexities, hindering the evaluation of modern Large Language Models (LLMs). Current benchmarks frequently focus on fine-grained code units, lack complete project context, and use inadequate evaluation metrics. To address these limitations, we introduce SWRBench , a new benchmark comprising 1000 manually verified Pull Requests (PRs) from GitHub, offering PR-centric review with full project context. SWRBench employs an objective LLM-based evaluation method that aligns strongly with human judgment (~90 agreement) by verifying if issues from a structured ground truth are covered in generated reviews. Our systematic evaluation of mainstream ACR tools and LLMs on SWRBench reveals that current systems underperform, and ACR tools are more adept at detecting functional errors. Subsequently, we propose and validate a simple multi-review aggregation strategy that significantly boosts ACR performance, increasing F1 scores by up to 43.67%. Our contributions include the SWRBench benchmark, its objective evaluation method, a comprehensive study of current ACR capabilities, and an effective enhancement approach, offering valuable insights for advancing ACR research.