mlsanityData to 14 Oct 2026

SemEval-2024 Shared Task 6: SHROOM, a Shared-task on Hallucinations and Related Observable Overgeneration Mistakes

arXiv 2403.07726 ↗First posted 12 Mar 2024cs.CLcited by 24

Not on the map. Why?

Abstract (arXiv)

This paper presents the results of the SHROOM, a shared task focused on detecting hallucinations: outputs from natural language generation (NLG) systems that are fluent, yet inaccurate. Such cases of overgeneration put in jeopardy many NLG applications, where correctness is often mission-critical. The shared task was conducted with a newly constructed dataset of 4000 model outputs labeled by 5 annotators each, spanning 3 NLP tasks: machine translation, paraphrase generation and definition modeling. The shared task was tackled by a total of 58 different users grouped in 42 teams, out of which 27 elected to write a system description paper; collectively, they submitted over 300 prediction sets on both tracks of the shared task. We observe a number of key trends in how this approach was tackled -- many participants rely on a handful of model, and often rely either on synthetic data for fine-tuning or zero-shot prompting strategies. While a majority of the teams did outperform our proposed baseline system, the performances of top-scoring systems are still consistent with a random handling of the more challenging items.

Key facts. Cited by 24 papers in the map (data through 14 Oct 2026). Builds on 24 of its 60 references that are in the map. First posted on arXiv 12 Mar 2024.

Builds on

24 of 60 references.

References · 24

  1. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  2. 2015A large annotated corpus for learning natural language inferencecited by 1,847
  3. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,312
  4. 2019PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarizationcited by 1,102
  5. 2021A Survey on Automated Fact-Checkingcited by 378
  6. 2016Diverse Beam Search: Decoding Diverse Solutions from Neural Sequence Modelscited by 301
  7. 2015A Neural Conversational Modelcited by 196
  8. 2021On Hallucination and Predictive Uncertainty in Conditional Language Generationcited by 176
  9. 2021The Curious Case of Hallucinations in Neural Machine Translationcited by 161
  10. 2020What Can We Learn from Collective Human Opinions on Natural Language Inference Data?cited by 153
  11. 2020Detecting Hallucinated Content in Conditional Neural Sequence Generationcited by 152
  12. 2021A Token-level Reference-free Hallucination Detection Benchmark for Free-form Text Generationcited by 138
12 more
  1. 2021Evaluating Attribution in Dialogue Systems: The BEGIN Benchmarkcited by 77
  2. 2022POTATO: The Portable Text Annotation Toolcited by 64
  3. 2016Definition Modeling: Learning to define word embeddings in natural languagecited by 46
  4. 2021Distributed NLI: Learning to Predict Human Opinion Distributions for Language Reasoningcited by 31
  5. 2018Open Subtitles Paraphrase Corpus for Six Languagescited by 24
  6. 2023Interpretable Word Sense Representations via Definition Generation: The Case of Semantic Change Analysiscited by 19
  7. 2021ProtAugment: Unsupervised diverse short-texts paraphrasing for intent detection meta-learningcited by 18
  8. 2022Semeval-2022 Task 1: CODWOE -- Comparing Dictionaries and Word Embeddingscited by 18
  9. 2023"Definition Modeling: To model definitions." Generating Definitions With Little to No Semanticscited by 4
  10. 2024MALTO at SemEval-2024 Task 6: Leveraging Synthetic Data for LLM Hallucination Detectioncited by 4
  11. 2024OPDAI at SemEval-2024 Task 6: Small LLMs can Accelerate Hallucination Detection with Weakly Supervised Datacited by 3
  12. 2023Why bother with geometry? On the relevance of linear decompositions of Transformer embeddingscited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 5; 2025: 10; 2026: 9.

24 papers.

Sort:

By area:

  1. 2024Knowledge Graphs, Large Language Models, and Hallucinations: An NLP Perspectivecited by 29
  2. 2025SemEval-2025 Task 3: Mu-SHROOM, the Multilingual Shared Task on Hallucinations and Related Observable Overgeneration Mistakescited by 27
  3. 2024FaithBench: A Diverse Hallucination Benchmark for Summarization by Modern LLMscited by 24
  4. 2026Can Humans Dream of Electric Sheep? Human-Written Samples for Fine-Grained Vision-and-Language Hallucination Benchmarkingcited by 4
  5. 2025HalluSearch at SemEval-2025 Task 3: A Search-Enhanced RAG Pipeline for Hallucination Detectioncited by 2
  6. 2025Can LLMs Detect Intrinsic Hallucinations in Paraphrasing and Machine Translation?cited by 2
  7. 2025MultiHal: Multilingual Dataset for Knowledge-Graph Grounded Evaluation of LLM Hallucinationscited by 2
  8. 2025Simplifications are Absolutists: How Simplified Language Reduces Word Sense Awareness in LLM-Generated Definitionscited by 2
  9. 2025Confabulations from ACL Publications (CAP): A Dataset for Scientific Hallucination Detectioncited by 2
  10. 2024SHROOM-INDElab at SemEval-2024 Task 6: Zero- and Few-Shot LLM-Based Classification for Hallucination Detectioncited by 1
  11. 2024The Two Sides of the Coin: Hallucination Generation and Detection with LLMs as Evaluators for LLMscited by 1
  12. 2025HausaNLP at SemEval-2025 Task 3: Towards a Fine-Grained Model-Aware Hallucination Detectioncited by 1
12 more
  1. 2026DSC2025 -- ViHallu Challenge: Detecting Hallucination in Vietnamese LLMscited by 1
  2. 2024AILS-NTUA at SemEval-2024 Task 6: Efficient model tuning for hallucination detection and analysiscited by 0
  3. 2025AILS-NTUA at SemEval-2025 Task 3: Leveraging Large Language Models and Translation Strategies for Multilingual Hallucination Detectioncited by 0
  4. 2025CCNU at SemEval-2025 Task 3: Leveraging Internal and External Knowledge of Large Language Models for Multilingual Hallucination Annotationcited by 0
  5. 2025Modelling Adjectival Modification Effects on Semantic Plausibilitycited by 0
  6. 2026Fabricator or dynamic translator?cited by 0
  7. 2026LLMs in the Real World: Evaluating "AI" in Emergency Contextscited by 0
  8. 2026Overview of SHROOM-Visions 2026: A Shared Task on Hallucination Detection in Large Vision-Language Modelscited by 0
  9. 2026SpanCalib-VLM: Calibrated Hallucination Span Detection in Vision-Language Modelscited by 0
  10. 2026SingProbe Technical Reportcited by 0
  11. 2026SKstars at SHROOM: Visions Agreement-Guided Ensembling of Zero-Shot and LoRA-Adapted Vision--Language Modelscited by 0
  12. 2026Halluscoring 2026: The first shared task on llms hallucination detection and answer verificationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023HaluEval: A Large-Scale Hallucination Evaluation Benchmark for Large Language Models
  2. 2024THaMES: An End-to-End Tool for Hallucination Mitigation and Evaluation in Large Language Models
  3. 2025From Detection to Diagnosis: Advancing Hallucination Analysis with Automated Data Synthesis
  4. 2026ToolHACE: A Dataset for Detecting Hallucinations in Tool-Based LLM Responses
  5. 2024Investigating the Role of Prompting and External Tools in Hallucination Rates of Large Language Models
  6. 2023RAGTruth: A Hallucination Corpus for Developing Trustworthy Retrieval-Augmented Language Models
  7. 2026Rethinking Hallucinations: Correctness, Consistency, and Prompt Multiplicity
  8. 2026PRISM: Probing Reasoning, Instruction, and Source Memory in LLM Hallucinations