mlsanityData to 7 Oct 2026

CRScore: Grounding Automated Evaluation of Code Review Comments in Code Claims and Smells

arXiv 2409.19801 ↗First posted 29 Sep 2024cs.SE cs.AI cs.CLcited by 8

Not on the map. Why?

Abstract (arXiv)

The task of automated code review has recently gained a lot of attention from the machine learning community. However, current review comment evaluation metrics rely on comparisons with a human-written reference for a given code change (also called a diff). Furthermore, code review is a one-to-many problem, like generation and summarization, with many "valid reviews" for a diff. Thus, we develop CRScore - a reference-free metric to measure dimensions of review quality like conciseness, comprehensiveness, and relevance. We design CRScore to evaluate reviews in a way that is grounded in claims and potential issues detected in the code by LLMs and static analyzers. We demonstrate that CRScore can produce valid, fine-grained scores of review quality that have the greatest alignment with human judgment among open source metrics (0.54 Spearman correlation) and are more sensitive than reference-based metrics. We also release a corpus of 2.9k human-annotated review quality scores for machine-generated and GitHub review comments to support the development of automated metrics.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Builds on 33 of its 78 references that are in the map. First posted on arXiv 29 Sep 2024.

Builds on

33 of 78 references.

References · 33

  1. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  3. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  4. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  5. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,388
  6. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  7. 2024DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligencecited by 1,154
  8. 2024LLM Evaluators Recognize and Favor Their Own Generationscited by 651
  9. 2020CodeBLEU: a Method for Automatic Evaluation of Code Synthesiscited by 468
  10. 2020USR: An Unsupervised and Reference Free Evaluation Metric for Dialog Generationcited by 186
  11. 2020Unsupervised Evaluation of Interactive Dialog with DialoGPTcited by 139
  12. 2023CodePlan: Repository-level Coding using LLMs and Planningcited by 139
21 more
  1. 2023CodeBERTScore: Evaluating Code Generation with Pretrained Models of Codecited by 101
  2. 2018ROUGE 2.0: Updated and Improved Measures for Evaluation of Summarization Taskscited by 66
  3. 2022Out of the BLEU: how should we assess quality of the Code Generation models?cited by 65
  4. 2022Using Pre-Trained Models to Boost Code Review Automationcited by 63
  5. 2021Towards Automating Code Review Activitiescited by 54
  6. 2019Does BLEU Score Work for Code Migration?cited by 49
  7. 2023LLaMA-Reviewer: Advancing Code Review Automation with Large Language Models through Parameter-Efficient Fine-Tuningcited by 47
  8. 2023InfoMetIC: An Informative Metric for Reference-free Image Caption Evaluationcited by 25
  9. 2024Fine-Tuning and Prompt Engineering for Large Language Models-based Code Review Automationcited by 23
  10. 2024AI-powered Code Review with LLMs: Early Resultscited by 22
  11. 2019VIFIDEL: Evaluating the Visual Fidelity of Image Descriptionscited by 21
  12. 2024Exploring the Capabilities of LLMs for Code Change Related Taskscited by 18
  13. 2024AI-Assisted Assessment of Coding Practices in Modern Code Reviewcited by 16
  14. 2018Predicting Usefulness of Code Review Comments using Textual Features and Developer Experiencecited by 13
  15. 2024Improving Automated Code Reviews: Learning from Experiencecited by 13
  16. 2023What Makes a Code Review Useful to OpenDev Developers? An Empirical Investigationcited by 10
  17. 2022Automated Identification of Toxic Code Reviews Using ToxiCRcited by 8
  18. 2020Towards evaluating and eliciting high-quality documentation for intelligent systemscited by 7
  19. 2024Modern Code Reviews -- Survey of Literature and Practicecited by 6
  20. 2024Prompt Learning for Multi-Label Code Smell Detection: A Promising Approachcited by 3
  21. 2024A GPT-based Code Review System for Programming Language Learningcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 6.

8 papers.

  1. 2026Code Review Agent Benchmarkcited by 4
  2. 2026SWE-Review: Closing the Loop on Issue Resolution with Agentic Code Reviewcited by 4
  3. 2025ConCISE: A Reference-Free Conciseness Evaluation Metric for LLM-Generated Answerscited by 1
  4. 2026ReFEree: Reference-Free and Fine-Grained Method for Evaluating Factual Consistency in Real-World Code Summarizationcited by 1
  5. 2026Understanding the Limits of Automated Evaluation for Code Review Bots in Practicecited by 1
  6. 2026Rethinking Code Review in the Age of AI: A Vision for Agentic Code Reviewcited by 1
  7. 2025MetaLint: Easy-to-Hard Generalization for Code Lintingcited by 0
  8. 2026CRJudgeBench: Can AI Detect Plausible but Invalid Code Reviews?cited by 0