mlsanityData to 14 Oct 2026

HalOmi: A Manually Annotated Benchmark for Multilingual Hallucination and Omission Detection in Machine Translation

arXiv 2305.11746 ↗First posted 19 May 2023cs.CLcited by 35

Not on the map. Why?

Abstract (arXiv)

Hallucinations in machine translation are translations that contain information completely unrelated to the input. Omissions are translations that do not include some of the input information. While both cases tend to be catastrophic errors undermining user trust, annotated data with these types of pathologies is extremely scarce and is limited to a few high-resource languages. In this work, we release an annotated dataset for the hallucination and omission phenomena covering 18 translation directions with varying resource levels and scripts. Our annotation covers different levels of partial and full hallucinations as well as omissions both at the sentence and at the word level. Additionally, we revisit previous methods for hallucination and omission detection, show that conclusions made based on a single language pair largely do not hold for a large-scale evaluation, and establish new solid baselines.

Key facts. Cited by 35 papers in the map (data through 14 Oct 2026). Builds on 28 of its 46 references that are in the map. First posted on arXiv 19 May 2023.

Builds on

28 of 46 references.

References · 28

  1. 2013Visualizing and Understanding Convolutional Networkscited by 2,290
  2. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,799
  3. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,312
  4. 2020COMET: A Neural Framework for MT Evaluationcited by 1,033
  5. 2019Attention is not Explanationcited by 929
  6. 2019Attention is not not Explanationcited by 636
  7. 2020Language-agnostic BERT Sentence Embeddingcited by 627
  8. 2019Is Attention Interpretable?cited by 389
  9. 2016Diverse Beam Search: Decoding Diverse Solutions from Neural Sequence Modelscited by 301
  10. 2016Understanding Neural Networks through Representation Erasurecited by 224
  11. 2021The Curious Case of Hallucinations in Neural Machine Translationcited by 161
  12. 2023Multilingual Machine Translation with Large Language Models: Empirical Results and Analysiscited by 160
16 more
  1. 2020Detecting Hallucinated Content in Conditional Neural Sequence Generationcited by 152
  2. 2020The elephant in the interpretability room: Why use attention as explanation when we have saliency methods?cited by 122
  3. 2019On NMT Search Errors and Model Errors: Cat Got Your Tongue?cited by 114
  4. 2022Detecting and Mitigating Hallucinations in Machine Translation: Model Internal Workings Alone Do Well, Sentence Similarity Even Bettercited by 75
  5. 2019Domain Robustness in Neural Machine Translationcited by 65
  6. 2022Looking for a Needle in a Haystack: A Comprehensive Study of Hallucinations in Neural Machine Translationcited by 61
  7. 2021Understanding the Properties of Minimum Bayes Risk Decoding in Neural Machine Translationcited by 56
  8. 2021Measuring and Increasing Context Usage in Context-Aware Machine Translationcited by 48
  9. 2022Bitext Mining Using Distilled Sentence Representations for Low-Resource Languagescited by 48
  10. 2022Towards Opening the Black Box of Neural Machine Translation: Source and Target Interpretations of the Transformercited by 45
  11. 2023Understanding and Detecting Hallucinations in Neural Machine Translation via Model Introspectioncited by 44
  12. 2016Representation of linguistic form and function in recurrent neural networkscited by 39
  13. 2019Naver Labs Europe's Systems for the WMT19 Machine Translation Robustness Taskcited by 23
  14. 2022Consistent Human Evaluation of Machine Translation across Language Pairscited by 13
  15. 2022Optimal Transport for Unsupervised Hallucination Detection in Neural Machine Translationcited by 13
  16. 2022As Little as Possible, as Much as Necessary: Detecting Over- and Undertranslations with Contrastive Conditioningcited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 7; 2024: 12; 2025: 10; 2026: 6.

35 papers.

Sort:

By area:

  1. 2023xCOMET: Transparent Machine Translation Evaluation through Fine-grained Error Detectioncited by 265
  2. 2024Information Flow Routes: Automatically Interpreting Language Models at Scalecited by 95
  3. 2023Cognitive Mirage: A Review of Hallucinations in Large Language Modelscited by 84
  4. 2024Multilingual Large Language Model: A Survey of Resources, Taxonomy and Frontierscited by 51
  5. 2025Large Language Models Hallucination: A Comprehensive Surveycited by 49
  6. 2023Improving Translation Faithfulness of Large Language Models via Augmenting Instructionscited by 28
  7. 2024A Probabilistic Framework for LLM Hallucination Detection via Belief Tree Propagationcited by 21
  8. 2024An Audit on the Perspectives and Challenges of Hallucinations in NLPcited by 17
  9. 2024Word Alignment as Preference for Machine Translationcited by 14
  10. 2024Teaching LLMs to Abstain across Languages via Multilingual Feedbackcited by 14
  11. 2023Why LLMs Hallucinate, and How to Get (Evidential) Closure: Perceptual, Intensional, and Extensional Learning for Faithful Natural Language Generationcited by 13
  12. 2024Machine Translation Hallucination Detection for Low and High Resource Languages using Large Language Modelscited by 13
23 more
  1. 2023AutoHall: Automated Factuality Hallucination Dataset Generation for Large Language Modelscited by 10
  2. 2024Enhanced Hallucination Detection in Neural Machine Translation through Simple Detector Aggregationcited by 9
  3. 2023Quantifying the Plausibility of Context Reliance in Neural Machine Translationcited by 7
  4. 2024A Survey of Automatic Hallucination Evaluation on Natural Language Generationcited by 7
  5. 2025CCHall: A Novel Benchmark for Joint Cross-Lingual and Cross-Modal Hallucinations Detection in Large Language Modelscited by 7
  6. 2023Insights into Classifying and Mitigating LLMs' Hallucinationscited by 6
  7. 2025Mitigating Hallucinated Translations in Large Language Models with Hallucination-focused Preference Optimizationcited by 6
  8. 2025A Unified Definition of Hallucination: It's The World Model, Stupid!cited by 6
  9. 2025How Much Do LLMs Hallucinate across Languages? On Realistic Multilingual Estimation of LLM Hallucinationcited by 5
  10. 2025Unsupervised Word-level Quality Estimation for Machine Translation Through the Lens of Annotators (Dis)agreementcited by 4
  11. 2025M2PO: Multi-Perspective Multi-Pair Preference Optimization for Machine Translationcited by 4
  12. 2024Towards Red Teaming in Multimodal and Multilingual Translationcited by 3
  13. 2025Conflicts in Texts: Data, Implications and Challengescited by 2
  14. 2026HalluScore: Large Language Model Hallucination Question Answering Benchmarkcited by 2
  15. 2026HalluTruthQA: A Fine-Grained Benchmark for Hallucination Detection, Localization, and Explanation in Arabic Question Answeringcited by 2
  16. 2024OTTAWA: Optimal TransporT Adaptive Word Aligner for Hallucination and Omission Translation Errors Detectioncited by 1
  17. 2024Analyzing Context Contributions in LLM-based Machine Translationcited by 1
  18. 2026Understanding Why Language Models Hallucinate: Testing Reasoning Against Priorscited by 1
  19. 2026Scaling Unsupervised Word Alignment to Documents via Structural Constraintscited by 1
  20. 2025AILS-NTUA at SemEval-2025 Task 3: Leveraging Large Language Models and Translation Strategies for Multilingual Hallucination Detectioncited by 0
  21. 2025Challenging Multilingual LLMs: A New Taxonomy and Benchmark for Unraveling Hallucination in Translationcited by 0
  22. 2026Fabricator or dynamic translator?cited by 0
  23. 2026Failure-Aware Long-Form Translation: Design and Implementation of a Recoverable LLM Translation Systemcited by 0