# DAHL: Domain-specific Automated Hallucination Evaluation of Long-Form Text through a Benchmark Dataset in Biomedicine

arXiv: https://arxiv.org/abs/2411.09255 · HTML: https://mlsanity.com/paper/2411.09255/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 36 references that are in the map.
- First posted on arXiv 14 Nov 2024.

## Builds on (most cited first, 29 of 29)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [TruthfulQA: Measuring How Models Mimic Human Falsehoods](https://mlsanity.com/paper/2109.07958/) (2021)
- [A Survey of Large Language Models](https://mlsanity.com/paper/2303.18223/) (2023)
- [Qwen2 Technical Report](https://mlsanity.com/paper/2407.10671/) (2024)
- [Gemma 2: Improving Open Language Models at a Practical Size](https://mlsanity.com/paper/2408.00118/) (2024)
- [Language Models (Mostly) Know What They Know](https://mlsanity.com/paper/2207.05221/) (2022)
- [Large Language Models Encode Clinical Knowledge](https://mlsanity.com/paper/2212.13138/) (2022)
- [Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?](https://mlsanity.com/paper/2202.12837/) (2022)
- [FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation](https://mlsanity.com/paper/2305.14251/) (2023)
- [Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Models](https://mlsanity.com/paper/2309.01219/) (2023)
- [Large Language Models Struggle to Learn Long-Tail Knowledge](https://mlsanity.com/paper/2211.08411/) (2022)
- [A Survey of Hallucination in Large Foundation Models](https://mlsanity.com/paper/2309.05922/) (2023)
- [Hallucination is Inevitable: An Innate Limitation of Large Language Models](https://mlsanity.com/paper/2401.11817/) (2024)
- [Do Prompt-Based Models Really Understand the Meaning of their Prompts?](https://mlsanity.com/paper/2109.01247/) (2021)
- [ChatDoctor: A Medical Chat Model Fine-Tuned on a Large Language Model Meta-AI (LLaMA) Using Medical Domain Knowledge](https://mlsanity.com/paper/2303.14070/) (2023)
- [Chain-of-Verification Reduces Hallucination in Large Language Models](https://mlsanity.com/paper/2309.11495/) (2023)
- [The Effect of Sampling Temperature on Problem Solving in Large Language Models](https://mlsanity.com/paper/2402.05201/) (2024)
- [Med-HALT: Medical Domain Hallucination Test for Large Language Models](https://mlsanity.com/paper/2307.15343/) (2023)
- [The Troubling Emergence of Hallucination in Large Language Models -- An Extensive Definition, Quantification, and Prescriptive Remediations](https://mlsanity.com/paper/2310.04988/) (2023)
- [A Survey of Large Language Models in Medicine: Progress, Application, and Challenge](https://mlsanity.com/paper/2311.05112/) (2023)
- [The Dawn After the Dark: An Empirical Study on Factuality Hallucination in Large Language Models](https://mlsanity.com/paper/2401.03205/) (2024)
- [FELM: Benchmarking Factuality Evaluation of Large Language Models](https://mlsanity.com/paper/2310.00741/) (2023)
- [Rethinking Explainability as a Dialogue: A Practitioner's Perspective](https://mlsanity.com/paper/2202.01875/) (2022)
- [Xiezhi: An Ever-Updating Benchmark for Holistic Domain Knowledge Evaluation](https://mlsanity.com/paper/2306.05783/) (2023)
- [Halo: Estimation and Reduction of Hallucinations in Open-Source Weak Large Language Models](https://mlsanity.com/paper/2308.11764/) (2023)
- [SemEval-2019 Task 8: Fact Checking in Community Question Answering Forums](https://mlsanity.com/paper/1906.01727/) (2019)
- [An Automatic Evaluation Framework for Multi-turn Medical Consultations Capabilities of Large Language Models](https://mlsanity.com/paper/2309.02077/) (2023)
- [AutoHall: Automated Factuality Hallucination Dataset Generation for Large Language Models](https://mlsanity.com/paper/2310.00259/) (2023)

## Built on by (most cited first, 4 of 4)

- [ProMedical: Hierarchical Fine-Grained Criteria Modeling for Medical LLM Alignment via Explicit Injection](https://mlsanity.com/paper/2604.08326/) (2026)
- [OAEI-LLM-T: A TBox Benchmark Dataset for Understanding Large Language Model Hallucinations in Ontology Matching](https://mlsanity.com/paper/2503.21813/) (2025)
- [Toward Reliable, Safe, and Secure LLMs for Scientific Applications](https://mlsanity.com/paper/2603.18235/) (2026)
- [CausalGaze: Unveiling Hallucinations via Counterfactual Graph Intervention in Large Language Models](https://mlsanity.com/paper/2604.11087/) (2026)

## Abstract (arXiv)

We introduce DAHL, a benchmark dataset and automated evaluation system designed to assess hallucination in long-form text generation, specifically within the biomedical domain. Our benchmark dataset, meticulously curated from biomedical research papers, consists of 8,573 questions across 29 categories. DAHL evaluates fact-conflicting hallucinations in Large Language Models (LLMs) by deconstructing responses into atomic units, each representing a single piece of information. The accuracy of these responses is averaged to produce the DAHL Score, offering a more in-depth evaluation of hallucinations compared to previous methods that rely on multiple-choice tasks. We conduct experiments with 8 different models, finding that larger models tend to hallucinate less; however, beyond a model size of 7 to 8 billion parameters, further scaling does not significantly improve factual accuracy. The DAHL Score holds potential as an efficient alternative to human-annotated preference labels, being able to be expanded to other specialized domains. We release the dataset and code in public.