mlsanityData to 7 Oct 2026

RovoDev Code Reviewer: A Large-Scale Online Evaluation of LLM-based Code Review Automation at Atlassian

arXiv 2601.01129 ↗First posted 3 Jan 2026cs.SE cs.AI cs.CLcited by 6

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLMs)-powered code review automation has the potential to transform code review workflows. Despite the advances of LLM-powered code review comment generation approaches, several practical challenges remain for designing enterprise-grade code review automation tools. In particular, this paper aims at answering the practical question: how can we design a review-guided, context-aware, quality-checked code review comment generation without fine-tuning? In this paper, we present RovoDev Code Reviewer, an enterprise-grade LLM-based code review automation tool designed and deployed at scale within Atlassian's development ecosystem with seamless integration into Atlassian's Bitbucket. Through the offline, online, user feedback evaluations over a one-year period, we conclude that RovoDev Code Reviewer is effective in generating code review comments that could lead to code resolution for 38.70% (i.e., comments that triggered code changes in the subsequent commits); and offers the promise of accelerating feedback cycles (i.e., decreasing the PR cycle time by 30.8%), alleviating reviewer workload (i.e., reducing the number of human-written comments by 35.6%), and improving overall software quality (i.e., finding errors with actionable suggestions).

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 24 of its 57 references that are in the map. First posted on arXiv 3 Jan 2026.

Builds on

24 of 57 references.

References · 24

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  2. 2022Survey of Hallucination in Natural Language Generationcited by 2,056
  3. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,582
  4. 2024A Survey on LLM-as-a-Judgecited by 1,384
  5. 2023SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Modelscited by 787
  6. 2024Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inferencecited by 527
  7. 2024From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judgecited by 480
  8. 2023FacTool: Factuality Detection in Generative AI -- A Tool Augmented Framework for Multi-Task and Multi-Domain Scenarioscited by 247
  9. 2022Automating Code Review Activities by Large-Scale Pre-trainingcited by 98
  10. 2024LLM Hallucinations in Practical Code Generation: Phenomena, Mechanism, and Mitigationcited by 92
  11. 2022Using Pre-Trained Models to Boost Code Review Automationcited by 63
  12. 2021Towards Automating Code Review Activitiescited by 54
12 more
  1. 2019On Learning Meaningful Code Changes via Neural Machine Translationcited by 50
  2. 2024Human-In-the-Loop Software Development Agentscited by 42
  3. 2024Fine-Tuning and Prompt Engineering for Large Language Models-based Code Review Automationcited by 23
  4. 2024Code Review Automation: Strengths and Weaknesses of the State of the Artcited by 20
  5. 2024AI-Assisted Assessment of Coding Practices in Modern Code Reviewcited by 16
  6. 2020Predicting Defective Lines Using a Model-Agnostic Techniquecited by 14
  7. 2021JITLine: A Simpler, Better, Faster, Finer-grained Just-In-Time Defect Predictioncited by 14
  8. 2025Too Noisy To Learn: Enhancing Data Quality for Code Review Comment Generationcited by 12
  9. 2023Pitfalls in Language Models for Code Intelligence: A Taxonomy and Surveycited by 9
  10. 2025What Types of Code Review Comments Do Developers Most Frequently Resolve?cited by 7
  11. 2025Retrieval-Augmented Code Review Comment Generationcited by 6
  12. 2021Assessing the Students' Understanding and their Mistakes in Code Review Checklists -- An Experience Report of 1,791 Code Review Checklist Questions from 394 Studentscited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 6.

6 papers.

  1. 2026HalluJudge: A Reference-Free Hallucination Detection for Context Misalignment in Code Review Automationcited by 5
  2. 2026Code Review Agent Benchmarkcited by 4
  3. 2026AgenticSCR: An Autonomous Agentic Secure Code Review for Immature Vulnerabilities Detectioncited by 2
  4. 2026SEVRA-BENCH: Social Engineering of Vulnerabilities in Review Agentscited by 1
  5. 2026Is Agentic Code Review Helpful? Mining Developers' Feedback to CodeRabbit Reviews in the Wildcited by 1
  6. 2026Using Agentic AI for contextualized and multifaceted code review at Ericssoncited by 0