# Benchmarking LLMs for Fine-Grained Code Review with Enriched Context in Practice

arXiv: https://arxiv.org/abs/2511.07017 · HTML: https://mlsanity.com/paper/2511.07017/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 50 references that are in the map.
- First posted on arXiv 10 Nov 2025.

## Builds on (most cited first, 17 of 17)

- [Learning to Compare: Relation Network for Few-Shot Learning](https://mlsanity.com/paper/1711.06025/) (2017)
- [Large Language Models for Software Engineering: A Systematic Literature Review](https://mlsanity.com/paper/2308.10620/) (2023)
- [A Survey on Large Language Models for Recommendation](https://mlsanity.com/paper/2305.19860/) (2023)
- [LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Tasks](https://mlsanity.com/paper/2406.18403/) (2024)
- [Trae Agent: An LLM-based Agent for Software Engineering with Test-time Scaling](https://mlsanity.com/paper/2507.23370/) (2025)
- [MarsCode Agent: AI-native Automated Bug Fixing](https://mlsanity.com/paper/2409.00899/) (2024)
- [ICE-Score: Instructing Large Language Models to Evaluate Code](https://mlsanity.com/paper/2304.14317/) (2023)
- [Repo2Run: Automated Building Executable Environment for Code Repository at Scale](https://mlsanity.com/paper/2502.13681/) (2025)
- [ReposVul: A Repository-Level High-Quality Vulnerability Dataset](https://mlsanity.com/paper/2401.13169/) (2024)
- [AUGER: Automatically Generating Review Comments with Pre-training Models](https://mlsanity.com/paper/2208.08014/) (2022)
- [AEGIS: An Agent-based Framework for General Bug Reproduction from Issue Descriptions](https://mlsanity.com/paper/2411.18015/) (2024)
- [RepoMasterEval: Evaluating Code Completion via Real-World Repositories](https://mlsanity.com/paper/2408.03519/) (2024)
- [Generation-based Code Review Automation: How Far Are We?](https://mlsanity.com/paper/2303.07221/) (2023)
- [Modern Code Reviews -- Survey of Literature and Practice](https://mlsanity.com/paper/2403.00088/) (2024)
- [CodeVisionary: An Agent-based Framework for Evaluating Large Language Models in Code Generation](https://mlsanity.com/paper/2504.13472/) (2025)
- [CORRECT: Code Reviewer Recommendation in GitHub Based on Cross-Project and Technology Experience](https://mlsanity.com/paper/1807.02965/) (2018)
- [Modeling Review History for Reviewer Recommendation:A Hypergraph Approach](https://mlsanity.com/paper/2204.09526/) (2022)

## Built on by (most cited first, 4 of 4)

- [AACR-Bench: Evaluating Automatic Code Review with Holistic Repository-Level Context](https://mlsanity.com/paper/2601.19494/) (2026)
- [Code Review Agent Benchmark](https://mlsanity.com/paper/2603.23448/) (2026)
- [Agentic Code Review in the Terminal: A Trajectory-Level Analysis of Behavior, Cost, and Human-Alignment](https://mlsanity.com/paper/2607.16740/) (2026)
- [Using Agentic AI for contextualized and multifaceted code review at Ericsson](https://mlsanity.com/paper/2609.15877/) (2026)

## Abstract (arXiv)

Code review is a cornerstone of software quality assurance, and recent advances in Large Language Models (LLMs) have shown promise in its automation. However, existing benchmarks for LLM-based code review face three major limitations. Lack of semantic context: most benchmarks provide only code diffs without textual information such as issue descriptions, which are crucial for understanding developer intent. Data quality issues: without rigorous validation, many samples are noisy-e.g., reviews on outdated or irrelevant code-reducing evaluation reliability. Coarse granularity: most benchmarks operate at the file or commit level, overlooking the fine-grained, line-level reasoning essential for precise review. We introduce ContextCRBench, a high-quality, context-rich benchmark for fine-grained LLM evaluation in code review. Our construction pipeline comprises: Raw Data Crawling, collecting 153.7K issues and pull requests from top-tier repositories; Comprehensive Context Extraction, linking issue-PR pairs for textual context and extracting the full surrounding function or class for code context; and Multi-stage Data Filtering, combining rule-based and LLM-based validation to remove outdated, malformed, or low-value samples, resulting in 67,910 context-enriched entries. ContextCRBench supports three evaluation scenarios aligned with the review workflow: hunk-level quality assessment, line-level defect localization, and line-level comment generation. Evaluating eight leading LLMs (four closed-source and four open-source) reveals that textual context yields greater performance gains than code context alone, while current LLMs remain far from human-level review ability. Deployed at ByteDance, ContextCRBench drives a self-evolving code review system, improving performance by 61.98% and demonstrating its robustness and industrial utility. https://github.com/kinesiatricssxilm14/ContextCRBench.