mlsanityData to 14 Oct 2026

HalluJudge: A Reference-Free Hallucination Detection for Context Misalignment in Code Review Automation

arXiv 2601.19072 ↗First posted 27 Jan 2026cs.SE cs.AIcited by 5

Not on the map. Why?

Abstract (arXiv)

Large Language models (LLMs) have shown strong capabilities in code review automation, such as review comment generation, yet they suffer from hallucinations -- where the generated review comments are ungrounded in the actual code -- poses a significant challenge to the adoption of LLMs in code review workflows. To address this, we explore effective and scalable methods for a hallucination detection in LLM-generated code review comments without the reference. In this work, we design HalluJudge that aims to assess the grounding of generated review comments based on the context alignment. HalluJudge includes four key strategies ranging from direct assessment to structured multi-branch reasoning (e.g., Tree-of-Thoughts). We conduct a comprehensive evaluation of these assessment strategies across Atlassian's enterprise-scale software projects to examine the effectiveness and cost-efficiency of HalluJudge. Furthermore, we analyze the alignment between HalluJudge's judgment and developer preference of the actual LLM-generated code review comments in the real-world production. Our results show that the hallucination assessment in HalluJudge is cost-effective with an F1 score of 0.85 and an average cost of 0.009. On average, 67% of the HalluJudge assessments are aligned with the developer preference of the actual LLM-generated review comments in the online production. Our results suggest that HalluJudge can serve as a practical safeguard to reduce developers' exposure to hallucinated comments, fostering trust in AI-assisted code reviews.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 29 of its 44 references that are in the map. First posted on arXiv 27 Jan 2026.

Builds on

29 of 44 references.

References · 29

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  4. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  5. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,211
  6. 2022Survey of Hallucination in Natural Language Generationcited by 2,075
  7. 2024A Survey on LLM-as-a-Judgecited by 1,423
  8. 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,282
  9. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 897
  10. 2022CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesiscited by 885
  11. 2022A Survey on In-context Learningcited by 721
  12. 2021Asleep at the Keyboard? Assessing the Security of GitHub Copilot's Code Contributionscited by 431
17 more
  1. 2023Large Language Models for Software Engineering: A Systematic Literature Reviewcited by 399
  2. 2023Larger language models do in-context learning differentlycited by 394
  3. 2022Using Pre-Trained Models to Boost Code Review Automationcited by 63
  4. 2021Towards Automating Code Review Activitiescited by 54
  5. 2024CodeHalu: Investigating Code Hallucinations in LLMs via Execution-based Verificationcited by 34
  6. 2024An Empirical Study of Static Analysis Tools for Secure Code Reviewcited by 17
  7. 2020Predicting Defective Lines Using a Model-Agnostic Techniquecited by 14
  8. 2024Improving Automated Code Reviews: Learning from Experiencecited by 13
  9. 2024DeepCRCEval: Revisiting the Evaluation of Code Review Comment Generationcited by 12
  10. 2025Too Noisy To Learn: Enhancing Data Quality for Code Review Comment Generationcited by 12
  11. 2023Pitfalls in Language Models for Code Intelligence: A Taxonomy and Surveycited by 9
  12. 2021A Systematic Literature Review and Taxonomy of Modern Code Reviewcited by 7
  13. 2025What Types of Code Review Comments Do Developers Most Frequently Resolve?cited by 7
  14. 2026RovoDev Code Reviewer: A Large-Scale Online Evaluation of LLM-based Code Review Automation at Atlassiancited by 6
  15. 2024ETF: An Entity Tracing Framework for Hallucination Detection in Code Summariescited by 5
  16. 2024Leveraging Reviewer Experience in Code Review Comment Generationcited by 4
  17. 2025Hallucinations in Code Change to Natural Language Generation: Prevalence and Evaluation of Detection Metricscited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 5.

5 papers.

  1. 2026Functional Entropy: Predicting Functional Correctness in LLM-Generated Code with Uncertainty Quantificationcited by 3
  2. 2026Understanding the Limits of Automated Evaluation for Code Review Bots in Practicecited by 1
  3. 2026Is Agentic Code Review Helpful? Mining Developers' Feedback to CodeRabbit Reviews in the Wildcited by 1
  4. 2026Agentic Code Review in the Terminal: A Trajectory-Level Analysis of Behavior, Cost, and Human-Alignmentcited by 1
  5. 2026Empirical Analysis and Detection of Hallucinations in LLM-Generated Bug Report Summariescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024De-Hallucinator: Mitigating LLM Hallucinations in Code Generation Tasks via Iterative Grounding
  2. 2026Rethinking Training Data for Generating Code Review Comments
  3. 2025CodeReviewQA: The Code Review Comprehension Assessment for Large Language Models
  4. 2024Prompting and Fine-tuning Large Language Models for Automated Code Review Comment Generation
  5. 2026Refusing the Impossible: A Taxonomy and Benchmark for Code Hallucination in Large Language Models ↗
  6. 2024LLM Hallucinations in Practical Code Generation: Phenomena, Mechanism, and Mitigation
  7. 2026Evaluating Inference-Time Defenses Against Package Hallucination in LLM-Generated Code
  8. 2025Large Language Model Critics for Execution-Free Evaluation of Code Changes