# HalluVerse25: Fine-grained Multilingual Benchmark Dataset for LLM Hallucinations

arXiv: https://arxiv.org/abs/2503.07833 · HTML: https://mlsanity.com/paper/2503.07833/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 22 references that are in the map.
- First posted on arXiv 10 Mar 2025.

## Builds on (most cited first, 16 of 16)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024)
- [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023)
- [Gemma 2: Improving Open Language Models at a Practical Size](https://mlsanity.com/paper/2408.00118/) (2024)
- [Phi-4 Technical Report](https://mlsanity.com/paper/2412.08905/) (2024)
- [INSIDE: LLMs' Internal States Retain the Power of Hallucination Detection](https://mlsanity.com/paper/2402.03744/) (2024)
- [A Stitch in Time Saves Nine: Detecting and Mitigating Hallucinations of LLMs by Validating Low-Confidence Generation](https://mlsanity.com/paper/2307.03987/) (2023)
- [Sources of Hallucination by Large Language Models on Inference Tasks](https://mlsanity.com/paper/2305.14552/) (2023)
- [Multilingual Machine Translation with Large Language Models: Empirical Results and Analysis](https://mlsanity.com/paper/2304.04675/) (2023)
- [Fine-grained Hallucination Detection and Editing for Language Models](https://mlsanity.com/paper/2401.06855/) (2024)
- [On Learning to Summarize with Large Language Models as References](https://mlsanity.com/paper/2305.14239/) (2023)
- [LLMs Will Always Hallucinate, and We Need to Live With This](https://mlsanity.com/paper/2409.05746/) (2024)
- [Diving Deep into Modes of Fact Hallucinations in Dialogue Systems](https://mlsanity.com/paper/2301.04449/) (2023)
- [Analyzing LLM Behavior in Dialogue Summarization: Unveiling Circumstantial Hallucination Trends](https://mlsanity.com/paper/2406.03487/) (2024)

## Built on by (most cited first, 14 of 14)

- [AraHalluEval: A Fine-grained Hallucination Evaluation Framework for Arabic LLMs](https://mlsanity.com/paper/2509.04656/) (2025)
- [Evaluating Arabic Large Language Models: A Survey of Benchmarks, Methods, and Gaps](https://mlsanity.com/paper/2510.13430/) (2025)
- [Do LLM hallucination detectors suffer from low-resource effect?](https://mlsanity.com/paper/2601.16766/) (2026)
- [PRISM: Probing Reasoning, Instruction, and Source Memory in LLM Hallucinations](https://mlsanity.com/paper/2604.16909/) (2026)
- [AraTable: Benchmarking LLMs' Reasoning and Understanding of Arabic Tabular Data](https://mlsanity.com/paper/2507.18442/) (2025)
- [HalluScore: Large Language Model Hallucination Question Answering Benchmark](https://mlsanity.com/paper/2605.17007/) (2026)
- [HalluTruthQA: A Fine-Grained Benchmark for Hallucination Detection, Localization, and Explanation in Arabic Question Answering](https://mlsanity.com/paper/2607.20219/) (2026)
- [ALPS: A Diagnostic Challenge Set for Arabic Linguistic & Pragmatic Reasoning](https://mlsanity.com/paper/2602.17054/) (2026)
- [BenHalluEval: A Multi-Task Hallucination Evaluation Framework for Large Language Models on Bengali](https://mlsanity.com/paper/2605.31483/) (2026)
- [Does Bielik Know What It Doesn't Know? Activation Dispersion Separates Entity Familiarity from Factual Reliability Across Model Scale](https://mlsanity.com/paper/2607.07670/) (2026)
- [Lost in Speech: Trilingual Spoken Hallucination Detection Across Audio and Transcripts](https://mlsanity.com/paper/2608.24707/) (2026)
- [FIBER: A Multilingual Evaluation Resource for Factual Inference Bias](https://mlsanity.com/paper/2512.11110/) (2025)
- [LongNovel: A Multi-Scale Benchmark for Hallucination Detection in Long-Context Novel Summarization](https://mlsanity.com/paper/2608.18082/) (2026)
- [Physics as the label for measuring and correcting materials reasoning in multimodal models](https://mlsanity.com/paper/2609.12181/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) are increasingly used in various contexts, yet remain prone to generating non-factual content, commonly referred to as "hallucinations". The literature categorizes hallucinations into several types, including entity-level, relation-level, and sentence-level hallucinations. However, existing hallucination datasets often fail to capture fine-grained hallucinations in multilingual settings. In this work, we introduce HalluVerse25, a multilingual LLM hallucination dataset that categorizes fine-grained hallucinations in English, Arabic, and Turkish. Our dataset construction pipeline uses an LLM to inject hallucinations into factual biographical sentences, followed by a rigorous human annotation process to ensure data quality. We evaluate several LLMs on HalluVerse25, providing valuable insights into how proprietary models perform in detecting LLM-generated hallucinations across different contexts.