mlsanityData to 7 Oct 2026

Comparing Hallucination Detection Metrics for Multilingual Generation

arXiv 2402.10496 ↗First posted 16 Feb 2024cs.CL cs.AIcited by 18

Not on the map. Why?

Abstract (arXiv)

While many hallucination detection techniques have been evaluated on English text, their effectiveness in multilingual contexts remains unknown. This paper assesses how well various factual hallucination detection metrics (lexical metrics like ROUGE and Named Entity Overlap, and Natural Language Inference (NLI)-based metrics) identify hallucinations in generated biographical summaries across languages. We compare how well automatic metrics correlate to each other and whether they agree with human judgments of factuality. Our analysis reveals that while the lexical metrics are ineffective, NLI-based metrics perform well, correlating with human annotations in many settings and often outperforming supervised models. However, NLI metrics are still limited, as they do not detect single-fact hallucinations well and fail for lower-resource languages. Therefore, our findings highlight the gaps in exisiting hallucination detection methods for non-English languages and motivate future research to develop more robust multilingual detection methods for LLM hallucinations.

Key facts. Cited by 18 papers in the map (data through 7 Oct 2026). Builds on 22 of its 34 references that are in the map. First posted on arXiv 16 Feb 2024.

Builds on

22 of 34 references.

References · 22

  1. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  2. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  3. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  4. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
  5. 2022Language Models (Mostly) Know What They Knowcited by 1,783
  6. 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,259
  7. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 913
  8. 2021SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarizationcited by 402
  9. 2023MEGA: Multilingual Evaluation of Generative AIcited by 291
  10. 2023FacTool: Factuality Detection in Generative AI -- A Tool Augmented Framework for Multi-Task and Multi-Domain Scenarioscited by 247
  11. 2020Reducing conversational agents' overconfidence through linguistic calibrationcited by 209
  12. 2023ChatGPT Beyond English: Towards a Comprehensive Evaluation of Large Language Models in Multilingual Learningcited by 203
10 more
  1. 2023A Stitch in Time Saves Nine: Detecting and Mitigating Hallucinations of LLMs by Validating Low-Confidence Generationcited by 202
  2. 2021The Curious Case of Hallucinations in Neural Machine Translationcited by 161
  3. 2021Entity-level Factual Consistency of Abstractive Text Summarizationcited by 118
  4. 2022Detecting and Mitigating Hallucinations in Machine Translation: Model Internal Workings Alone Do Well, Sentence Similarity Even Bettercited by 75
  5. 2023Investigating the Translation Performance of a Large Multilingual Language Model: the Case of BLOOMcited by 58
  6. 2023Detecting and Mitigating Hallucinations in Multilingual Summarisationcited by 48
  7. 2023Understanding and Detecting Hallucinations in Neural Machine Translation via Model Introspectioncited by 44
  8. 2022mFACE: Multilingual Summarization with Factual Consistency Evaluationcited by 42
  9. 2023SEAHORSE: A Multilingual, Multifaceted Dataset for Summarization Evaluationcited by 42
  10. 2023Correction with Backtracking Reduces Hallucination in Summarizationcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 8; 2025: 9; 2026: 1.

18 papers.

Sort:

By area:

  1. 2024Decoding by Contrasting Knowledge: Enhancing LLMs' Confidence on Edited Factscited by 36
  2. 2024CaLMQA: Exploring culturally specific long-form question answering across 23 languagescited by 31
  3. 2024Knowledge Graphs, Large Language Models, and Hallucinations: An NLP Perspectivecited by 29
  4. 2025The Illusion of Progress: Re-evaluating Hallucination Detection in LLMscited by 22
  5. 2024Multi-FAct: Assessing Factuality of Multilingual LLMs using FActScorecited by 18
  6. 2024Teaching LLMs to Abstain across Languages via Multilingual Feedbackcited by 14
  7. 2024On the Limitations of Large Language Models (LLMs): False Attributioncited by 12
  8. 2025CausalAbstain: Enhancing Multilingual LLMs with Causal Reasoning for Trustworthy Abstentioncited by 10
  9. 2024A Survey of Automatic Hallucination Evaluation on Natural Language Generationcited by 7
  10. 2024Multilingual Hallucination Gaps in Large Language Modelscited by 7
  11. 2025Trustworthy Machine Learning via Memorization and the Granular Long-Tail: A Survey on Interactions, Tradeoffs, and Beyondcited by 6
  12. 2025Evaluating Evaluation Metrics -- The Mirage of Hallucination Detectioncited by 6
6 more
  1. 2026IslamicLegalBench: Evaluating LLMs Knowledge and Reasoning of Islamic Law Across 1,200 Years of Islamic Pluralist Legal Traditionscited by 5
  2. 2025The Illusionist's Prompt: Exposing the Factual Vulnerabilities of Large Language Models with Linguistic Nuancescited by 2
  3. 2025Can LLMs Detect Intrinsic Hallucinations in Paraphrasing and Machine Translation?cited by 2
  4. 2025Beyond English: Unveiling Multilingual Bias in LLM Copyright Compliancecited by 1
  5. 2025MSA at SemEval-2025 Task 3: High Quality Weak Labeling and LLM Ensemble Verification for Multilingual Hallucination Detectioncited by 1
  6. 2025AILS-NTUA at SemEval-2025 Task 3: Leveraging Large Language Models and Translation Strategies for Multilingual Hallucination Detectioncited by 0