# HalluHard: A Hard Multi-Turn Hallucination Benchmark

arXiv: https://arxiv.org/abs/2602.01031 · HTML: https://mlsanity.com/paper/2602.01031/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 57 references that are in the map.
- First posted on arXiv 1 Feb 2026.

## Builds on (most cited first, 19 of 19)

- [TruthfulQA: Measuring How Models Mimic Human Falsehoods](https://mlsanity.com/paper/2109.07958/) (2021)
- [FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation](https://mlsanity.com/paper/2305.14251/) (2023)
- [SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models](https://mlsanity.com/paper/2303.08896/) (2023)
- [Evaluating the Factual Consistency of Abstractive Text Summarization](https://mlsanity.com/paper/1910.12840/) (2019)
- [Measuring short-form factuality in large language models](https://mlsanity.com/paper/2411.04368/) (2024)
- [Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools](https://mlsanity.com/paper/2405.20362/) (2024)
- [Long-form factuality in large language models](https://mlsanity.com/paper/2403.18802/) (2024)
- [AbstentionBench: Reasoning LLMs Fail on Unanswerable Questions](https://mlsanity.com/paper/2506.09038/) (2025)
- [Do Language Models Know When They're Hallucinating References?](https://mlsanity.com/paper/2305.18248/) (2023)
- [Winning Gold at IMO 2025 with a Model-Agnostic Verification-and-Refinement Pipeline](https://mlsanity.com/paper/2507.15855/) (2025)
- [The Illusion of Diminishing Returns: Measuring Long Horizon Execution in LLMs](https://mlsanity.com/paper/2509.09677/) (2025)
- [FLAME: Factuality-Aware Alignment for Large Language Models](https://mlsanity.com/paper/2405.01525/) (2024)
- [MedHallu: A Comprehensive Benchmark for Detecting Medical Hallucinations in Large Language Models](https://mlsanity.com/paper/2502.14302/) (2025)
- [WildHallucinations: Evaluating Long-form Factuality in LLMs with Real-World Entity Queries](https://mlsanity.com/paper/2407.17468/) (2024)
- [FaithBench: A Diverse Hallucination Benchmark for Summarization by Modern LLMs](https://mlsanity.com/paper/2410.13210/) (2024)
- [Importing Phantoms: Measuring LLM Package Hallucination Vulnerabilities](https://mlsanity.com/paper/2501.19012/) (2025)
- [Precise Information Control in Long-Form Text Generation](https://mlsanity.com/paper/2506.06589/) (2025)
- [Towards Transparent Reasoning: What Drives Faithfulness in Large Language Models?](https://mlsanity.com/paper/2510.24236/) (2025)
- [KnowMT-Bench: Benchmarking Knowledge-Intensive Long-Form Question Answering in Multi-Turn Dialogues](https://mlsanity.com/paper/2509.21856/) (2025)

## Built on by (most cited first, 9 of 9)

- [Who Checks the Citations? Benchmarking Legal Hallucination Detection](https://mlsanity.com/paper/2606.21155/) (2026)
- [HALLMARK: How to Diagnose LLM Citation Verifiers and When to Deploy Them](https://mlsanity.com/paper/2607.18360/) (2026)
- [RankJudge: A Multi-Turn LLM-as-a-Judge Synthetic Benchmark Generator](https://mlsanity.com/paper/2605.21748/) (2026)
- [The Future of Facts: Tracing the Factual Generation-Verification Gap](https://mlsanity.com/paper/2605.27564/) (2026)
- [Conformal Language Modeling via Posterior Sampling](https://mlsanity.com/paper/2606.03731/) (2026)
- [How Do Uncertainty Estimators Align with Hallucination Occurrence in LLMs?](https://mlsanity.com/paper/2605.27016/) (2026)
- [CORTEX: Token-Level Hallucination Detection in RAG via Comparative Internal Representations](https://mlsanity.com/paper/2606.31033/) (2026)
- [Tracing the Cascade: A Topology-Aware Evaluation Framework for Scientific Agent Hallucinations](https://mlsanity.com/paper/2608.00711/) (2026)
- [Epistemic Policy Divergence in Multi-Turn LLM Contamination: A Protocol-Gradient Investigation](https://mlsanity.com/paper/2609.35308/) (2026)

## Abstract (arXiv)

Large language models (LLMs) still produce plausible-sounding but ungrounded factual claims, a problem that worsens in multi-turn dialogue as context grows and early errors cascade. We introduce HalluHard, a challenging multi-turn hallucination benchmark with 950 seed questions spanning four high-stakes domains: legal cases, research questions, medical guidelines, and coding. We operationalize groundedness by requiring inline citations for factual assertions. To support reliable evaluation in open-ended settings, we propose a judging pipeline that iteratively retrieves evidence via web search. It can fetch, filter, and parse full-text sources (including PDFs) to assess whether cited material actually supports the generated content. Across a diverse set of frontier proprietary and open-weight models, hallucinations remain substantial even with web search (≈ 30% for the strongest configuration, Opus-4.5 with web search), with content-grounding errors persisting at high rates. Finally, we show that hallucination behavior is shaped by model capacity, turn position, effective reasoning, and the type of knowledge required.