mlsanityData to 14 Oct 2026

SemEval-2025 Task 3: Mu-SHROOM, the Multilingual Shared Task on Hallucinations and Related Observable Overgeneration Mistakes

arXiv 2504.11975 ↗First posted 16 Apr 2025cs.CLcited by 27

Not on the map. Why?

Abstract (arXiv)

We present the Mu-SHROOM shared task which is focused on detecting hallucinations and other overgeneration mistakes in the output of instruction-tuned large language models (LLMs). Mu-SHROOM addresses general-purpose LLMs in 14 languages, and frames the hallucination detection problem as a span-labeling task. We received 2,618 submissions from 43 participating teams employing diverse methodologies. The large number of submissions underscores the interest of the community in hallucination detection. We present the results of the participating systems and conduct an empirical analysis to identify key factors contributing to strong performance in this task. We also emphasize relevant current challenges, notably the varying degree of hallucinations across languages and the high annotator disagreement when labeling hallucination spans.

Key facts. Cited by 27 papers in the map (data through 14 Oct 2026). Builds on 20 of its 59 references that are in the map. First posted on arXiv 16 Apr 2025.

Builds on

20 of 59 references.

References · 20

  1. 2024ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Toolscited by 1,121
  2. 2023How Language Model Hallucinations Can Snowballcited by 311
  3. 2023RAGTruth: A Hallucination Corpus for Developing Trustworthy Retrieval-Augmented Language Modelscited by 283
  4. 2023OpenChat: Advancing Open-source Language Models with Mixed-Quality Datacited by 252
  5. 2015A Neural Conversational Modelcited by 196
  6. 2021The Curious Case of Hallucinations in Neural Machine Translationcited by 161
  7. 2020Detecting Hallucinated Content in Conditional Neural Sequence Generationcited by 152
  8. 2024Fine-grained Hallucination Detection and Editing for Language Modelscited by 149
  9. 2021A Token-level Reference-free Hallucination Detection Benchmark for Free-form Text Generationcited by 138
  10. 2023The Troubling Emergence of Hallucination in Large Language Models -- An Extensive Definition, Quantification, and Prescriptive Remediationscited by 131
  11. 2023Factuality Challenges in the Era of Large Language Modelscited by 126
  12. 2024Beyond Functional Correctness: Exploring Hallucinations in LLM-Generated Codecited by 107
8 more
  1. 2022Stop Measuring Calibration When Humans Disagreecited by 71
  2. 2024CroissantLLM: A Truly Bilingual French-English Language Modelcited by 53
  3. 2023FactCHD: Benchmarking Fact-Conflicting Hallucination Detectioncited by 29
  4. 2024SemEval-2024 Shared Task 6: SHROOM, a Shared-task on Hallucinations and Related Observable Overgeneration Mistakescited by 24
  5. 2024Poro 34B and the Blessing of Multilingualitycited by 18
  6. 2025HalluSearch at SemEval-2025 Task 3: A Search-Enhanced RAG Pipeline for Hallucination Detectioncited by 2
  7. 2025NCL-UoR at SemEval-2025 Task 3: Detecting Multilingual Hallucination and Related Observable Overgeneration Text Spans with Modified RefChecker and Modified SeflCheckGPTcited by 1
  8. 2025HausaNLP at SemEval-2025 Task 3: Towards a Fine-Grained Model-Aware Hallucination Detectioncited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 5; 2026: 22.

27 papers.

Sort:

By area:

  1. 2025Large Language Models Hallucination: A Comprehensive Surveycited by 49
  2. 2025When Models Lie, We Learn: Multilingual Span-Level Hallucination Detection with PsiloQAcited by 14
  3. 2026Do LLM hallucination detectors suffer from low-resource effect?cited by 4
  4. 2026Beyond Document Grounding: Span-Level Hallucination Detection over Code, Tool Output, and Documentscited by 3
  5. 2025Confabulations from ACL Publications (CAP): A Dataset for Scientific Hallucination Detectioncited by 2
  6. 2026HalluScore: Large Language Model Hallucination Question Answering Benchmarkcited by 2
  7. 2025InterpDetect: Interpretable Signals for Detecting Hallucinations in Retrieval-Augmented Generationcited by 1
  8. 2026DSC2025 -- ViHallu Challenge: Detecting Hallucination in Vietnamese LLMscited by 1
  9. 2026Does Bielik Know What It Doesn't Know? Activation Dispersion Separates Entity Familiarity from Factual Reliability Across Model Scalecited by 1
  10. 2026HalluTruthQA-4K: A Fine-Grained Corpus and Annotation Process for Arabic Hallucination Detection and Truth Verificationcited by 1
  11. 2026Lost in Speech: Trilingual Spoken Hallucination Detection Across Audio and Transcriptscited by 1
  12. 2025TUM-MiKaNi at SemEval-2025 Task 3: Towards Multilingual and Knowledge-Aware Non-factual Hallucination Identificationcited by 0
15 more
  1. 2026Lowest Span Confidence: Zero-Shot Hallucination Detection from a Single LLM Responsecited by 0
  2. 2026Fabricator or dynamic translator?cited by 0
  3. 2026Rethinking Evaluation for LLM Hallucination Detection: A Desiderata, A New RAG-based Benchmark, New Insightscited by 0
  4. 2026Analyzing the Correlation Between Hallucinations and Knowledge Conflicts in Large Language Modelscited by 0
  5. 2026X-MADAM-RAG: Diagnosing and Handling Chinese-English Evidence Conflict in Retrieval-Augmented Generationcited by 0
  6. 2026SpanUQ: Span-Level Uncertainty Quantification for Large Language Model Generationcited by 0
  7. 2026Graded Entity-Familiarity Readouts in Language Models: Polish Adaptation, Cross-Language Robustness, and Refusal Steeringcited by 0
  8. 2026Symbolic Augmentation Closes a Canonical-Equivalence Blind Spot in Neural Fact-Checkerscited by 0
  9. 2026Kernel Token Contradiction: a Fast and Principled Approach for LLM Claim Uncertainty Quantificationcited by 0
  10. 2026Overview of SHROOM-Visions 2026: A Shared Task on Hallucination Detection in Large Vision-Language Modelscited by 0
  11. 2026SpanCalib-VLM: Calibrated Hallucination Span Detection in Vision-Language Modelscited by 0
  12. 2026Enoki: Efficient Multi-Level Hallucination Detectioncited by 0
  13. 2026Two-Token Features and Small-Large Ensembles for VLM Hallucination Detectioncited by 0
  14. 2026SKstars at SHROOM: Visions Agreement-Guided Ensembling of Zero-Shot and LoRA-Adapted Vision--Language Modelscited by 0
  15. 2026Halluscoring 2026: The first shared task on llms hallucination detection and answer verificationcited by 0