# SemEval-2024 Shared Task 6: SHROOM, a Shared-task on Hallucinations and Related Observable Overgeneration Mistakes

arXiv: https://arxiv.org/abs/2403.07726 · HTML: https://mlsanity.com/paper/2403.07726/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 60 references that are in the map.
- First posted on arXiv 12 Mar 2024.

## Builds on (most cited first, 24 of 24)

- [TruthfulQA: Measuring How Models Mimic Human Falsehoods](https://mlsanity.com/paper/2109.07958/) (2021)
- [A large annotated corpus for learning natural language inference](https://mlsanity.com/paper/1508.05326/) (2015)
- [No Language Left Behind: Scaling Human-Centered Machine Translation](https://mlsanity.com/paper/2207.04672/) (2022)
- [PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization](https://mlsanity.com/paper/1912.08777/) (2019)
- [A Survey on Automated Fact-Checking](https://mlsanity.com/paper/2108.11896/) (2021)
- [Diverse Beam Search: Decoding Diverse Solutions from Neural Sequence Models](https://mlsanity.com/paper/1610.02424/) (2016)
- [A Neural Conversational Model](https://mlsanity.com/paper/1506.05869/) (2015)
- [On Hallucination and Predictive Uncertainty in Conditional Language Generation](https://mlsanity.com/paper/2103.15025/) (2021)
- [The Curious Case of Hallucinations in Neural Machine Translation](https://mlsanity.com/paper/2104.06683/) (2021)
- [What Can We Learn from Collective Human Opinions on Natural Language Inference Data?](https://mlsanity.com/paper/2010.03532/) (2020)
- [Detecting Hallucinated Content in Conditional Neural Sequence Generation](https://mlsanity.com/paper/2011.02593/) (2020)
- [A Token-level Reference-free Hallucination Detection Benchmark for Free-form Text Generation](https://mlsanity.com/paper/2104.08704/) (2021)
- [Evaluating Attribution in Dialogue Systems: The BEGIN Benchmark](https://mlsanity.com/paper/2105.00071/) (2021)
- [POTATO: The Portable Text Annotation Tool](https://mlsanity.com/paper/2212.08620/) (2022)
- [Definition Modeling: Learning to define word embeddings in natural language](https://mlsanity.com/paper/1612.00394/) (2016)
- [Distributed NLI: Learning to Predict Human Opinion Distributions for Language Reasoning](https://mlsanity.com/paper/2104.08676/) (2021)
- [Open Subtitles Paraphrase Corpus for Six Languages](https://mlsanity.com/paper/1809.06142/) (2018)
- [Interpretable Word Sense Representations via Definition Generation: The Case of Semantic Change Analysis](https://mlsanity.com/paper/2305.11993/) (2023)
- [ProtAugment: Unsupervised diverse short-texts paraphrasing for intent detection meta-learning](https://mlsanity.com/paper/2105.12995/) (2021)
- [Semeval-2022 Task 1: CODWOE -- Comparing Dictionaries and Word Embeddings](https://mlsanity.com/paper/2205.13858/) (2022)
- ["Definition Modeling: To model definitions." Generating Definitions With Little to No Semantics](https://mlsanity.com/paper/2306.08433/) (2023)
- [MALTO at SemEval-2024 Task 6: Leveraging Synthetic Data for LLM Hallucination Detection](https://mlsanity.com/paper/2403.00964/) (2024)
- [OPDAI at SemEval-2024 Task 6: Small LLMs can Accelerate Hallucination Detection with Weakly Supervised Data](https://mlsanity.com/paper/2402.12913/) (2024)
- [Why bother with geometry? On the relevance of linear decompositions of Transformer embeddings](https://mlsanity.com/paper/2310.06977/) (2023)

## Built on by (most cited first, 24 of 24)

- [Knowledge Graphs, Large Language Models, and Hallucinations: An NLP Perspective](https://mlsanity.com/paper/2411.14258/) (2024)
- [SemEval-2025 Task 3: Mu-SHROOM, the Multilingual Shared Task on Hallucinations and Related Observable Overgeneration Mistakes](https://mlsanity.com/paper/2504.11975/) (2025)
- [FaithBench: A Diverse Hallucination Benchmark for Summarization by Modern LLMs](https://mlsanity.com/paper/2410.13210/) (2024)
- [Can Humans Dream of Electric Sheep? Human-Written Samples for Fine-Grained Vision-and-Language Hallucination Benchmarking](https://mlsanity.com/paper/2608.01021/) (2026)
- [HalluSearch at SemEval-2025 Task 3: A Search-Enhanced RAG Pipeline for Hallucination Detection](https://mlsanity.com/paper/2504.10168/) (2025)
- [Can LLMs Detect Intrinsic Hallucinations in Paraphrasing and Machine Translation?](https://mlsanity.com/paper/2504.20699/) (2025)
- [MultiHal: Multilingual Dataset for Knowledge-Graph Grounded Evaluation of LLM Hallucinations](https://mlsanity.com/paper/2505.14101/) (2025)
- [Simplifications are Absolutists: How Simplified Language Reduces Word Sense Awareness in LLM-Generated Definitions](https://mlsanity.com/paper/2507.11981/) (2025)
- [Confabulations from ACL Publications (CAP): A Dataset for Scientific Hallucination Detection](https://mlsanity.com/paper/2510.22395/) (2025)
- [SHROOM-INDElab at SemEval-2024 Task 6: Zero- and Few-Shot LLM-Based Classification for Hallucination Detection](https://mlsanity.com/paper/2404.03732/) (2024)
- [The Two Sides of the Coin: Hallucination Generation and Detection with LLMs as Evaluators for LLMs](https://mlsanity.com/paper/2407.09152/) (2024)
- [HausaNLP at SemEval-2025 Task 3: Towards a Fine-Grained Model-Aware Hallucination Detection](https://mlsanity.com/paper/2503.19650/) (2025)
- [DSC2025 -- ViHallu Challenge: Detecting Hallucination in Vietnamese LLMs](https://mlsanity.com/paper/2601.04711/) (2026)
- [AILS-NTUA at SemEval-2024 Task 6: Efficient model tuning for hallucination detection and analysis](https://mlsanity.com/paper/2404.01210/) (2024)
- [AILS-NTUA at SemEval-2025 Task 3: Leveraging Large Language Models and Translation Strategies for Multilingual Hallucination Detection](https://mlsanity.com/paper/2503.02442/) (2025)
- [CCNU at SemEval-2025 Task 3: Leveraging Internal and External Knowledge of Large Language Models for Multilingual Hallucination Annotation](https://mlsanity.com/paper/2505.11965/) (2025)
- [Modelling Adjectival Modification Effects on Semantic Plausibility](https://mlsanity.com/paper/2507.21828/) (2025)
- [Fabricator or dynamic translator?](https://mlsanity.com/paper/2604.15165/) (2026)
- [LLMs in the Real World: Evaluating "AI" in Emergency Contexts](https://mlsanity.com/paper/2607.00019/) (2026)
- [Overview of SHROOM-Visions 2026: A Shared Task on Hallucination Detection in Large Vision-Language Models](https://mlsanity.com/paper/2608.25662/) (2026)
- [SpanCalib-VLM: Calibrated Hallucination Span Detection in Vision-Language Models](https://mlsanity.com/paper/2608.29974/) (2026)
- [SingProbe Technical Report](https://mlsanity.com/paper/2608.30703/) (2026)
- [SKstars at SHROOM: Visions Agreement-Guided Ensembling of Zero-Shot and LoRA-Adapted Vision--Language Models](https://mlsanity.com/paper/2609.24198/) (2026)
- [Halluscoring 2026: The first shared task on llms hallucination detection and answer verification](https://mlsanity.com/paper/2609.38355/) (2026)

## Abstract (arXiv)

This paper presents the results of the SHROOM, a shared task focused on detecting hallucinations: outputs from natural language generation (NLG) systems that are fluent, yet inaccurate. Such cases of overgeneration put in jeopardy many NLG applications, where correctness is often mission-critical. The shared task was conducted with a newly constructed dataset of 4000 model outputs labeled by 5 annotators each, spanning 3 NLP tasks: machine translation, paraphrase generation and definition modeling. The shared task was tackled by a total of 58 different users grouped in 42 teams, out of which 27 elected to write a system description paper; collectively, they submitted over 300 prediction sets on both tracks of the shared task. We observe a number of key trends in how this approach was tackled -- many participants rely on a handful of model, and often rely either on synthetic data for fine-tuning or zero-shot prompting strategies. While a majority of the teams did outperform our proposed baseline system, the performances of top-scoring systems are still consistent with a random handling of the more challenging items.