Leveraging Reward Models for Guiding Code Review Comment Generation
On the map: LLM alignment / RLHF / preference optimisation → Process reward models for reasoning
Abstract (arXiv)
Code review is a crucial component of modern software development, involving the evaluation of code quality, providing feedback on potential issues, and refining the code to address identified problems. Despite these benefits, code review can be rather time consuming, and influenced by subjectivity and human factors. For these reasons, techniques to (partially) automate the code review process have been proposed in the literature. Among those, the ones exploiting deep learning (DL) are able to tackle the generative aspect of code review, by commenting on a given code as a human reviewer would do (i.e., comment generation task) or by automatically implementing code changes required to address a reviewer's comment (i.e., code refinement task). In this paper, we introduce CoRAL, a deep learning framework automating review comment generation by exploiting reinforcement learning with a reward mechanism considering both the semantics of the generated comments as well as their usefulness as input for other models automating the code refinement task. The core idea is that if the DL model generates comments that are semantically similar to the expected ones or can be successfully implemented by a second model specialized in code refinement, these comments are likely to be meaningful and useful, thus deserving a high reward in the reinforcement learning framework. We present both quantitative and qualitative comparisons between the comments generated by CoRAL and those produced by the latest baseline techniques, highlighting the effectiveness and superiority of our approach.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Process reward models for reasoning, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 37 of its 85 references that are in the map. First posted on arXiv 4 Jun 2025.
Builds on
37 of 85 references.
References · 37
- 2022Training language models to follow instructions with human feedbackcited by 15,712
- 2017Proximal Policy Optimization Algorithmscited by 14,441
- 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
- 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
- 2015Distilling the Knowledge in a Neural Networkcited by 10,076
- 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
- 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
- 2022Generative Adversarial Networkscited by 5,603
- 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
- 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
- 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
- 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
25 more
- 2023Code Llama: Open Foundation Models for Codecited by 2,105
- 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,084
- 2022Scaling Laws for Reward Model Overoptimizationcited by 1,056
- 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 774
- 2022Improving alignment of dialogue agents via targeted human judgementscited by 527
- 2019Emergent Tool Use From Multi-Agent Autocurriculacited by 308
- 2017StarCraft II: A New Challenge for Reinforcement Learningcited by 268
- 2021Comparing Kullback-Leibler Divergence and Mean Squared Error Loss in Knowledge Distillationcited by 131
- 2018Improving Automatic Source Code Summarization via Deep Reinforcement Learningcited by 99
- 2022Automating Code Review Activities by Large-Scale Pre-trainingcited by 98
- 2019Adaptive Stress Testing for Autonomous Vehiclescited by 70
- 2022Using Pre-Trained Models to Boost Code Review Automationcited by 63
- 2024CodeUltraFeedback: An LLM-as-a-Judge Dataset for Aligning Large Language Models to Coding Preferencescited by 56
- 2021Towards Automating Code Review Activitiescited by 54
- 2021Augmenting Automated Game Testing with Deep Reinforcement Learningcited by 38
- 2017Continuous Integration, Delivery and Deployment: A Systematic Review on Approaches, Tools, Challenges and Practicescited by 22
- 2019Automated Video Game Testing Using Synthetic and Human-Like Agentscited by 22
- 2019CORE: Automating Review Recommendation for Code Changescited by 14
- 2022Using Reinforcement Learning for Load Testing of Video Gamescited by 10
- 2021Breaking Down Multilingual Machine Translationcited by 8
- 2016Learning a Static Analyzer from Datacited by 7
- 2024Improving the Learning of Code Review Successive Tasks with Cross-Task Knowledge Distillationcited by 7
- 2023Code Reviewer Recommendation for Architecture Violations: An Exploratory Studycited by 3
- 2024Enhancing Software Effort Estimation through Reinforcement Learning-based Project Management-Oriented Feature Selectioncited by 2
- 2020Towards Automated Safety Coverage and Testing for Autonomous Vehicles with Reinforcement Learningcited by 1
Built on by
Citing papers per year: 2025: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2026Rethinking Training Data for Generating Code Review Comments
- 2025LAURA: Enhancing Code Review Generation with Context-Enriched Retrieval-Augmented LLM
- 2025Retrieval-Augmented Code Review Comment Generation
- 2026From Code Review to Code Critique: Intent, Drift, and Spotlight for AI-Generated Diffs at Scale
- 2026Automated Classification of Human Code Review Comments with Large Language Models
- 2026"Go Home Copilot, You're Drunk": Understanding Developer Responses to Agent-Generated Code Review Comments
- 2026Rethinking Code Review in the Age of AI: A Vision for Agentic Code Review
- 2025CodeReviewQA: The Code Review Comprehension Assessment for Large Language Models