# CodeReviewQA: The Code Review Comprehension Assessment for Large Language Models

arXiv: https://arxiv.org/abs/2503.16167 · HTML: https://mlsanity.com/paper/2503.16167/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 41 references that are in the map.
- First posted on arXiv 20 Mar 2025.

## Builds on (most cited first, 22 of 22)

- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [Program Synthesis with Large Language Models](https://mlsanity.com/paper/2108.07732/) (2021)
- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions](https://mlsanity.com/paper/2406.15877/) (2024)
- [Leveraging Large Language Models for Multiple Choice Question Answering](https://mlsanity.com/paper/2210.12353/) (2022)
- [Automating Code Review Activities by Large-Scale Pre-training](https://mlsanity.com/paper/2203.09095/) (2022)
- [Using Pre-Trained Models to Boost Code Review Automation](https://mlsanity.com/paper/2201.06850/) (2022)
- [Towards Automating Code Review Activities](https://mlsanity.com/paper/2101.02518/) (2021)
- [On Learning Meaningful Code Changes via Neural Machine Translation](https://mlsanity.com/paper/1901.09102/) (2019)
- [Exploring the Potential of ChatGPT in Automated Code Refinement: An Empirical Study](https://mlsanity.com/paper/2309.08221/) (2023)
- [Automatically Generating Commit Messages from Diffs using Neural Machine Translation](https://mlsanity.com/paper/1708.09492/) (2017)
- [LLMs May Perform MCQA by Selecting the Least Incorrect Option](https://mlsanity.com/paper/2402.01349/) (2024)
- [Sampling in Software Engineering Research: A Critical Review and Guidelines](https://mlsanity.com/paper/2002.07764/) (2020)
- [Fine-Tuning and Prompt Engineering for Large Language Models-based Code Review Automation](https://mlsanity.com/paper/2402.00905/) (2024)
- [Code Review Automation: Strengths and Weaknesses of the State of the Art](https://mlsanity.com/paper/2401.05136/) (2024)
- [Predicting Usefulness of Code Review Comments using Textual Features and Developer Experience](https://mlsanity.com/paper/1807.04485/) (2018)
- [Automatic Generation of Pull Request Descriptions](https://mlsanity.com/paper/1909.06987/) (2019)
- [Improving Automated Code Reviews: Learning from Experience](https://mlsanity.com/paper/2402.03777/) (2024)
- [Too Noisy To Learn: Enhancing Data Quality for Code Review Comment Generation](https://mlsanity.com/paper/2502.02757/) (2025)
- [Code Review Comments: Language Matters](https://mlsanity.com/paper/1803.02205/) (2018)

## Built on by (most cited first, 3 of 3)

- [Beyond Code Snippets: Benchmarking LLMs on Repository-Level Question Answering](https://mlsanity.com/paper/2603.26567/) (2026)
- [Enhancing Large Language Models with Retrieval Augmented Generation for Software Testing and Inspection Automation](https://mlsanity.com/paper/2604.15270/) (2026)
- [From Static to Dynamic: Benchmarking Real-World Code Review with MCR-Bench](https://mlsanity.com/paper/2608.27442/) (2026)

## Abstract (arXiv)

State-of-the-art large language models (LLMs) have demonstrated impressive code generation capabilities but struggle with real-world software engineering tasks, such as revising source code to address code reviews, hindering their practical use. Code review comments are often implicit, ambiguous, and colloquial, requiring models to grasp both code and human intent. This challenge calls for evaluating large language models' ability to bridge both technical and conversational contexts. While existing work has employed the automated code refinement (ACR) task to resolve these comments, current evaluation methods fall short, relying on text matching metrics that provide limited insight into model failures and remain susceptible to training data contamination. To address these limitations, we introduce a novel evaluation benchmark, CodeReviewQA that enables us to conduct fine-grained assessment of model capabilities and mitigate data contamination risks. In CodeReviewQA, we decompose the generation task of code refinement into three essential reasoning steps: textitchange type recognition (CTR), textitchange localisation (CL), and textitsolution identification (SI). Each step is reformulated as multiple-choice questions with varied difficulty levels, enabling precise assessment of model capabilities, while mitigating data contamination risks. Our comprehensive evaluation spans 72 recently released large language models on 900 manually curated, high-quality examples across nine programming languages. Our results show that CodeReviewQA is able to expose specific model weaknesses in code review comprehension, disentangled from their generative automated code refinement results.