# Halu-J: Critique-Based Hallucination Judge

arXiv: https://arxiv.org/abs/2407.12943 · HTML: https://mlsanity.com/paper/2407.12943/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 64 references that are in the map.
- First posted on arXiv 17 Jul 2024.

## Builds on (most cited first, 26 of 26)

- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://mlsanity.com/paper/2205.14135/) (2022)
- [FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning](https://mlsanity.com/paper/2307.08691/) (2023)
- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023)
- [FEVER: a large-scale dataset for Fact Extraction and VERification](https://mlsanity.com/paper/1803.05355/) (2018)
- [ZeRO: Memory Optimizations Toward Training Trillion Parameter Models](https://mlsanity.com/paper/1910.02054/) (2019)
- [FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation](https://mlsanity.com/paper/2305.14251/) (2023)
- [Training Deep Nets with Sublinear Memory Cost](https://mlsanity.com/paper/1604.06174/) (2016)
- [Large Language Models Can Be Easily Distracted by Irrelevant Context](https://mlsanity.com/paper/2302.00093/) (2023)
- [Adversarial NLI: A New Benchmark for Natural Language Understanding](https://mlsanity.com/paper/1910.14599/) (2019)
- [AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback](https://mlsanity.com/paper/2305.14387/) (2023)
- [Self-critiquing models for assisting human evaluators](https://mlsanity.com/paper/2206.05802/) (2022)
- [A Survey on Automated Fact-Checking](https://mlsanity.com/paper/2108.11896/) (2021)
- [ZeRO-Offload: Democratizing Billion-Scale Model Training](https://mlsanity.com/paper/2101.06840/) (2021)
- [PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimization](https://mlsanity.com/paper/2306.05087/) (2023)
- [RAGTruth: A Hallucination Corpus for Developing Trustworthy Retrieval-Augmented Language Models](https://mlsanity.com/paper/2401.00396/) (2023)
- [MiniCheck: Efficient Fact-Checking of LLMs on Grounding Documents](https://mlsanity.com/paper/2404.10774/) (2024)
- [FacTool: Factuality Detection in Generative AI -- A Tool Augmented Framework for Multi-Task and Multi-Domain Scenarios](https://mlsanity.com/paper/2307.13528/) (2023)
- [WANLI: Worker and AI Collaboration for Natural Language Inference Dataset Creation](https://mlsanity.com/paper/2201.05955/) (2022)
- [Generative Judge for Evaluating Alignment](https://mlsanity.com/paper/2310.05470/) (2023)
- [Fine-grained Hallucination Detection and Editing for Language Models](https://mlsanity.com/paper/2401.06855/) (2024)
- [WiCE: Real-World Entailment for Claims in Wikipedia](https://mlsanity.com/paper/2303.01432/) (2023)
- [Can Large Language Models be Trusted for Evaluation? Scalable Meta-Evaluation of LLMs as Evaluators via Agent Debate](https://mlsanity.com/paper/2401.16788/) (2024)
- [DocLens: Multi-aspect Fine-grained Evaluation for Medical Text Generation](https://mlsanity.com/paper/2311.09581/) (2023)
- [Mitigating Hallucinations in Large Language Models via Self-Refinement-Enhanced Knowledge Retrieval](https://mlsanity.com/paper/2405.06545/) (2024)

## Built on by (most cited first, 14 of 14)

- [From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judge](https://mlsanity.com/paper/2411.16594/) (2024)
- [LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methods](https://mlsanity.com/paper/2412.05579/) (2024)
- [Hunyuan-TurboS: Advancing Large Language Models through Mamba-Transformer Synergy and Adaptive Chain-of-Thought](https://mlsanity.com/paper/2505.15431/) (2025)
- [Toward Generalizable Evaluation in the LLM Era: A Survey Beyond Benchmarks](https://mlsanity.com/paper/2504.18838/) (2025)
- [A Comprehensive Survey of Direct Preference Optimization: Datasets, Theories, Variants, and Applications](https://mlsanity.com/paper/2410.15595/) (2024)
- [Does Context Matter? ContextualJudgeBench for Evaluating LLM-based Judges in Contextual Settings](https://mlsanity.com/paper/2503.15620/) (2025)
- [Credible Plan-Driven RAG Method for Multi-Hop Question Answering](https://mlsanity.com/paper/2504.16787/) (2025)
- [An Investigation into Value Misalignment in LLM-Generated Texts for Cultural Heritage](https://mlsanity.com/paper/2501.02039/) (2025)
- [Verdict: A Library for Scaling Judge-Time Compute](https://mlsanity.com/paper/2502.18018/) (2025)
- [Multi-Agent LLM Judge: automatic personalized LLM judge design for evaluating natural language generation applications](https://mlsanity.com/paper/2504.02867/) (2025)
- [FaithLens: Detecting and Explaining Faithfulness Hallucination](https://mlsanity.com/paper/2512.20182/) (2025)
- [Interpreting LLM-as-a-Judge Policies via Verifiable Global Explanations](https://mlsanity.com/paper/2510.08120/) (2025)
- [Global Consistency Repair for Natural-Language Claim Sets](https://mlsanity.com/paper/2601.13600/) (2026)
- [DIAL-SUMMER: A Structured Evaluation Framework of Hierarchical Errors in Dialogue Summaries](https://mlsanity.com/paper/2602.08149/) (2026)

## Abstract (arXiv)

Large language models (LLMs) frequently generate non-factual content, known as hallucinations. Existing retrieval-augmented-based hallucination detection approaches typically address this by framing it as a classification task, evaluating hallucinations based on their consistency with retrieved evidence. However, this approach usually lacks detailed explanations for these evaluations and does not assess the reliability of these explanations. Furthermore, deficiencies in retrieval systems can lead to irrelevant or partially relevant evidence retrieval, impairing the detection process. Moreover, while real-world hallucination detection requires analyzing multiple pieces of evidence, current systems usually treat all evidence uniformly without considering its relevance to the content. To address these challenges, we introduce Halu-J, a critique-based hallucination judge with 7 billion parameters. Halu-J enhances hallucination detection by selecting pertinent evidence and providing detailed critiques. Our experiments indicate that Halu-J outperforms GPT-4o in multiple-evidence hallucination detection and matches its capability in critique generation and evidence selection. We also introduce ME-FEVER, a new dataset designed for multiple-evidence hallucination detection. Our code and dataset can be found in https://github.com/GAIR-NLP/factool .