Understanding the Limits of Automated Evaluation for Code Review Bots in Practice
Not on the map. Why?
Abstract (arXiv)
Automated code review (ACR) bots are increasingly used in industrial software development to assist developers during pull request (PR) review. As adoption grows, a key challenge is how to evaluate the usefulness of bot-generated comments reliably and at scale. In practice, such evaluation often relies on developer actions and annotations that are shaped by contextual and organizational factors, complicating their use as objective ground truth. We examine the feasibility and limitations of automating the evaluation of LLM-powered ACR bots in an industrial setting. We analyze an industrial dataset from Beko comprising 2,604 bot-generated PR comments, each labeled by software engineers as fixed/wontFix. Two automated evaluation approaches, G-Eval and an LLM-as-a-Judge pipeline, are applied using both binary decisions and a 0-4 Likert-scale formulation, enabling a controlled comparison against developer-provided labels. Across Gemini-2.5-pro, GPT-4.1-mini, and GPT-5.2, both evaluation strategies achieve only moderate alignment with human labels. Agreement ratios range from approximately 0.44 to 0.62, with noticeable variation across models and between binary and Likert-scale formulations, indicating sensitivity to both model choice and evaluation design. Our findings highlight practical limitations in fully automating the evaluation of ACR bot comments in industrial contexts. Developer actions such as resolving or ignoring comments reflect not only comment quality, but also contextual constraints, prioritization decisions, and workflow dynamics that are difficult to capture through static artifacts. Insights from a follow-up interview with a software engineering director further corroborate that developer labeling behavior is strongly influenced by workflow pressures and organizational constraints, reinforcing the challenges of treating such signals as objective ground truth.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 18 of its 41 references that are in the map. First posted on arXiv 27 Apr 2026.
Builds on
18 of 41 references.
References · 18
- 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
- 2023Benchmarking Large Language Models for News Summarizationcited by 444
- 2023Ragas: Automated Evaluation of Retrieval Augmented Generationcited by 408
- 2024Humans or LLMs as the Judge? A Study on Judgement Biasescited by 314
- 2024LLM-based NLG Evaluation: Current Status and Challengescited by 71
- 2025Can LLMs Replace Human Evaluators? An Empirical Study of LLM-as-a-Judge in Software Engineeringcited by 64
- 2023LLaMA-Reviewer: Advancing Code Review Automation with Large Language Models through Parameter-Efficient Fine-Tuningcited by 47
- 2023Exploring the Potential of ChatGPT in Automated Code Refinement: An Empirical Studycited by 44
- 2024Fine-Tuning and Prompt Engineering for Large Language Models-based Code Review Automationcited by 23
- 2025BitsAI-CR: Automated Code Review via LLM in Practicecited by 22
- 2024AI-Assisted Assessment of Coding Practices in Modern Code Reviewcited by 16
- 2024CRScore: Grounding Automated Evaluation of Code Review Comments in Code Claims and Smellscited by 8
6 more
- 2025LAURA: Enhancing Code Review Generation with Context-Enriched Retrieval-Augmented LLMcited by 8
- 2025Evaluating Large Language Models for Code Reviewcited by 7
- 2026HalluJudge: A Reference-Free Hallucination Detection for Context Misalignment in Code Review Automationcited by 5
- 2025Think Again! The Effect of Test-Time Compute on Preferences, Opinions, and Beliefs of Large Language Modelscited by 2
- 2026A Survey of Code Review Benchmarks and Evaluation Practices in Pre-LLM and LLM Eracited by 2
- 2025Towards Trustworthy Sentiment Analysis in Software Engineering: Dataset Characteristics and Tool Selectioncited by 1
Built on by
Citing papers per year: 2026: 1.
1 paper.