# SemEval-2025 Task 3: Mu-SHROOM, the Multilingual Shared Task on Hallucinations and Related Observable Overgeneration Mistakes

arXiv: https://arxiv.org/abs/2504.11975 · HTML: https://mlsanity.com/paper/2504.11975/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 59 references that are in the map.
- First posted on arXiv 16 Apr 2025.

## Builds on (most cited first, 20 of 20)

- [ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Tools](https://mlsanity.com/paper/2406.12793/) (2024)
- [How Language Model Hallucinations Can Snowball](https://mlsanity.com/paper/2305.13534/) (2023)
- [RAGTruth: A Hallucination Corpus for Developing Trustworthy Retrieval-Augmented Language Models](https://mlsanity.com/paper/2401.00396/) (2023)
- [OpenChat: Advancing Open-source Language Models with Mixed-Quality Data](https://mlsanity.com/paper/2309.11235/) (2023)
- [A Neural Conversational Model](https://mlsanity.com/paper/1506.05869/) (2015)
- [The Curious Case of Hallucinations in Neural Machine Translation](https://mlsanity.com/paper/2104.06683/) (2021)
- [Detecting Hallucinated Content in Conditional Neural Sequence Generation](https://mlsanity.com/paper/2011.02593/) (2020)
- [Fine-grained Hallucination Detection and Editing for Language Models](https://mlsanity.com/paper/2401.06855/) (2024)
- [A Token-level Reference-free Hallucination Detection Benchmark for Free-form Text Generation](https://mlsanity.com/paper/2104.08704/) (2021)
- [The Troubling Emergence of Hallucination in Large Language Models -- An Extensive Definition, Quantification, and Prescriptive Remediations](https://mlsanity.com/paper/2310.04988/) (2023)
- [Factuality Challenges in the Era of Large Language Models](https://mlsanity.com/paper/2310.05189/) (2023)
- [Beyond Functional Correctness: Exploring Hallucinations in LLM-Generated Code](https://mlsanity.com/paper/2404.00971/) (2024)
- [Stop Measuring Calibration When Humans Disagree](https://mlsanity.com/paper/2210.16133/) (2022)
- [CroissantLLM: A Truly Bilingual French-English Language Model](https://mlsanity.com/paper/2402.00786/) (2024)
- [FactCHD: Benchmarking Fact-Conflicting Hallucination Detection](https://mlsanity.com/paper/2310.12086/) (2023)
- [SemEval-2024 Shared Task 6: SHROOM, a Shared-task on Hallucinations and Related Observable Overgeneration Mistakes](https://mlsanity.com/paper/2403.07726/) (2024)
- [Poro 34B and the Blessing of Multilinguality](https://mlsanity.com/paper/2404.01856/) (2024)
- [HalluSearch at SemEval-2025 Task 3: A Search-Enhanced RAG Pipeline for Hallucination Detection](https://mlsanity.com/paper/2504.10168/) (2025)
- [NCL-UoR at SemEval-2025 Task 3: Detecting Multilingual Hallucination and Related Observable Overgeneration Text Spans with Modified RefChecker and Modified SeflCheckGPT](https://mlsanity.com/paper/2503.01921/) (2025)
- [HausaNLP at SemEval-2025 Task 3: Towards a Fine-Grained Model-Aware Hallucination Detection](https://mlsanity.com/paper/2503.19650/) (2025)

## Built on by (most cited first, 27 of 27)

- [Large Language Models Hallucination: A Comprehensive Survey](https://mlsanity.com/paper/2510.06265/) (2025)
- [When Models Lie, We Learn: Multilingual Span-Level Hallucination Detection with PsiloQA](https://mlsanity.com/paper/2510.04849/) (2025)
- [Do LLM hallucination detectors suffer from low-resource effect?](https://mlsanity.com/paper/2601.16766/) (2026)
- [Beyond Document Grounding: Span-Level Hallucination Detection over Code, Tool Output, and Documents](https://mlsanity.com/paper/2607.00895/) (2026)
- [Confabulations from ACL Publications (CAP): A Dataset for Scientific Hallucination Detection](https://mlsanity.com/paper/2510.22395/) (2025)
- [HalluScore: Large Language Model Hallucination Question Answering Benchmark](https://mlsanity.com/paper/2605.17007/) (2026)
- [InterpDetect: Interpretable Signals for Detecting Hallucinations in Retrieval-Augmented Generation](https://mlsanity.com/paper/2510.21538/) (2025)
- [DSC2025 -- ViHallu Challenge: Detecting Hallucination in Vietnamese LLMs](https://mlsanity.com/paper/2601.04711/) (2026)
- [Does Bielik Know What It Doesn't Know? Activation Dispersion Separates Entity Familiarity from Factual Reliability Across Model Scale](https://mlsanity.com/paper/2607.07670/) (2026)
- [HalluTruthQA-4K: A Fine-Grained Corpus and Annotation Process for Arabic Hallucination Detection and Truth Verification](https://mlsanity.com/paper/2608.03966/) (2026)
- [Lost in Speech: Trilingual Spoken Hallucination Detection Across Audio and Transcripts](https://mlsanity.com/paper/2608.24707/) (2026)
- [TUM-MiKaNi at SemEval-2025 Task 3: Towards Multilingual and Knowledge-Aware Non-factual Hallucination Identification](https://mlsanity.com/paper/2507.00579/) (2025)
- [Lowest Span Confidence: Zero-Shot Hallucination Detection from a Single LLM Response](https://mlsanity.com/paper/2601.19918/) (2026)
- [Fabricator or dynamic translator?](https://mlsanity.com/paper/2604.15165/) (2026)
- [Rethinking Evaluation for LLM Hallucination Detection: A Desiderata, A New RAG-based Benchmark, New Insights](https://mlsanity.com/paper/2605.11330/) (2026)
- [Analyzing the Correlation Between Hallucinations and Knowledge Conflicts in Large Language Models](https://mlsanity.com/paper/2606.08705/) (2026)
- [X-MADAM-RAG: Diagnosing and Handling Chinese-English Evidence Conflict in Retrieval-Augmented Generation](https://mlsanity.com/paper/2606.12903/) (2026)
- [SpanUQ: Span-Level Uncertainty Quantification for Large Language Model Generation](https://mlsanity.com/paper/2607.05721/) (2026)
- [Graded Entity-Familiarity Readouts in Language Models: Polish Adaptation, Cross-Language Robustness, and Refusal Steering](https://mlsanity.com/paper/2607.13568/) (2026)
- [Symbolic Augmentation Closes a Canonical-Equivalence Blind Spot in Neural Fact-Checkers](https://mlsanity.com/paper/2607.16212/) (2026)
- [Kernel Token Contradiction: a Fast and Principled Approach for LLM Claim Uncertainty Quantification](https://mlsanity.com/paper/2608.22506/) (2026)
- [Overview of SHROOM-Visions 2026: A Shared Task on Hallucination Detection in Large Vision-Language Models](https://mlsanity.com/paper/2608.25662/) (2026)
- [SpanCalib-VLM: Calibrated Hallucination Span Detection in Vision-Language Models](https://mlsanity.com/paper/2608.29974/) (2026)
- [Enoki: Efficient Multi-Level Hallucination Detection](https://mlsanity.com/paper/2609.00581/) (2026)
- [Two-Token Features and Small-Large Ensembles for VLM Hallucination Detection](https://mlsanity.com/paper/2609.10244/) (2026)
- [SKstars at SHROOM: Visions Agreement-Guided Ensembling of Zero-Shot and LoRA-Adapted Vision--Language Models](https://mlsanity.com/paper/2609.24198/) (2026)
- [Halluscoring 2026: The first shared task on llms hallucination detection and answer verification](https://mlsanity.com/paper/2609.38355/) (2026)

## Abstract (arXiv)

We present the Mu-SHROOM shared task which is focused on detecting hallucinations and other overgeneration mistakes in the output of instruction-tuned large language models (LLMs). Mu-SHROOM addresses general-purpose LLMs in 14 languages, and frames the hallucination detection problem as a span-labeling task. We received 2,618 submissions from 43 participating teams employing diverse methodologies. The large number of submissions underscores the interest of the community in hallucination detection. We present the results of the participating systems and conduct an empirical analysis to identify key factors contributing to strong performance in this task. We also emphasize relevant current challenges, notably the varying degree of hallucinations across languages and the high annotator disagreement when labeling hallucination spans.