mlsanityData to 7 Oct 2026

Looking for a Needle in a Haystack: A Comprehensive Study of Hallucinations in Neural Machine Translation

arXiv 2208.05309 ↗First posted 10 Aug 2022cs.CL cs.LGcited by 60

Not on the map. Why?

Abstract (arXiv)

Although the problem of hallucinations in neural machine translation (NMT) has received some attention, research on this highly pathological phenomenon lacks solid ground. Previous work has been limited in several ways: it often resorts to artificial settings where the problem is amplified, it disregards some (common) types of hallucinations, and it does not validate adequacy of detection heuristics. In this paper, we set foundations for the study of NMT hallucinations. First, we work in a natural setting, i.e., in-domain data without artificial noise neither in training nor in inference. Next, we annotate a dataset of over 3.4k sentences indicating different kinds of critical errors and hallucinations. Then, we turn to detection methods and both revisit methods used previously and propose using glass-box uncertainty-based detectors. Overall, we show that for preventive settings, (i) previously used methods are largely inadequate, (ii) sequence log-probability works best and performs on par with reference-based methods. Finally, we propose DeHallucinator, a simple method for alleviating hallucinations at test time that significantly reduces the hallucinatory rate. To ease future research, we release our annotated dataset for WMT18 German-English data, along with the model, training data, and code.

Key facts. Cited by 60 papers in the map (data through 7 Oct 2026). Builds on 41 of its 63 references that are in the map. First posted on arXiv 10 Aug 2022.

Builds on

41 of 63 references.

References · 41

  1. 2017Attention Is All You Needcited by 59,932
  2. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  3. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  4. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,416
  5. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  6. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  7. 2022Survey of Hallucination in Natural Language Generationcited by 2,056
  8. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  9. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
  10. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  11. 2020COMET: A Neural Framework for MT Evaluationcited by 1,024
  12. 2019Attention is not Explanationcited by 916
29 more
  1. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 844
  2. 2021Deduplicating Training Data Makes Language Models Bettercited by 705
  3. 2019Attention is not not Explanationcited by 629
  4. 2019Is Attention Interpretable?cited by 387
  5. 2017Six Challenges for Neural Machine Translationcited by 334
  6. 2021Quality at a Glance: An Audit of Web-Crawled Multilingual Datasetscited by 241
  7. 2020Unsupervised Quality Estimation for Neural Machine Translationcited by 210
  8. 2021The Curious Case of Hallucinations in Neural Machine Translationcited by 161
  9. 2021To Ship or Not to Ship: An Extensive Evaluation of Automatic Metrics for Machine Translationcited by 161
  10. 2020Detecting Hallucinated Content in Conditional Neural Sequence Generationcited by 151
  11. 2020On Exposure Bias, Hallucination and Domain Shift in Neural Machine Translationcited by 127
  12. 2020The elephant in the interpretability room: Why use attention as explanation when we have saliency methods?cited by 122
  13. 2019On NMT Search Errors and Model Errors: Cat Got Your Tongue?cited by 114
  14. 2019Learning to Deceive with Attention-Based Explanationscited by 111
  15. 2021High Quality Rather than High Model Probability: Minimum Bayes Risk Decoding with Neural Metricscited by 104
  16. 2018Context-Aware Neural Machine Translation Learns Anaphora Resolutioncited by 97
  17. 2019Attention Interpretability Across NLP Taskscited by 71
  18. 2020Analyzing the Source and Target Contributions to Predictions in Neural Machine Translationcited by 71
  19. 2019Calibration of Encoder Decoder Models for Neural Machine Translationcited by 67
  20. 2019Domain Robustness in Neural Machine Translationcited by 64
  21. 2021Facebook AI WMT21 News Translation Task Submissioncited by 57
  22. 2020Unbabel's Participation in the WMT20 Metrics Shared Taskcited by 56
  23. 2021Understanding the Properties of Minimum Bayes Risk Decoding in Neural Machine Translationcited by 56
  24. 2022Quality-Aware Decoding for Neural Machine Translationcited by 49
  25. 2022Identifying Weaknesses in Machine Translation Metrics Through Minimum Bayes Risk Decoding: A Case Study for COMETcited by 41
  26. 2022SALTED: A Framework for SAlient Long-Tail Translation Error Detectioncited by 23
  27. 2019Naver Labs Europe's Systems for the WMT19 Machine Translation Robustness Taskcited by 22
  28. 2021DEEP: DEnoising Entity Pre-training for Neural Machine Translationcited by 15
  29. 2022Probing Causes of Hallucinations in Neural Machine Translationscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 4; 2023: 18; 2024: 13; 2025: 19; 2026: 6.

60 papers.

Sort:

By area:

  1. 2023Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Modelscited by 695
  2. 2024Hallucination is Inevitable: An Innate Limitation of Large Language Modelscited by 329
  3. 2023xCOMET: Transparent Machine Translation Evaluation through Fine-grained Error Detectioncited by 260
  4. 2024LLMs Know More Than They Show: On the Intrinsic Representation of LLM Hallucinationscited by 230
  5. 2023Hallucinations in Large Multilingual Translation Modelscited by 126
  6. 2022ChatGPT Makes Medicine Easy to Swallow: An Exploratory Case Study on Simplified Radiology Reportscited by 100
  7. 2024HaloScope: Harnessing Unlabeled LLM Generations for Hallucination Detectioncited by 96
  8. 2024Model Tells You Where to Merge: Adaptive KV Cache Merging for LLMs on Long-Context Taskscited by 94
  9. 2023Cognitive Mirage: A Review of Hallucinations in Large Language Modelscited by 83
  10. 2022Detecting and Mitigating Hallucinations in Machine Translation: Model Internal Workings Alone Do Well, Sentence Similarity Even Bettercited by 75
  11. 2023Uncertainty in Natural Language Generation: From Theory to Applicationscited by 61
  12. 2023Leveraging GPT-4 for Automatic Translation Post-Editingcited by 56
38 more
  1. 2023On Early Detection of Hallucinations in Factual Question Answeringcited by 54
  2. 2025Large Language Models Hallucination: A Comprehensive Surveycited by 45
  3. 2023Understanding and Detecting Hallucinations in Neural Machine Translation via Model Introspectioncited by 44
  4. 2023Mitigating Hallucinations and Off-target Machine Translation with Source-Contrastive and Language-Contrastive Decodingcited by 42
  5. 2023HalOmi: A Manually Annotated Benchmark for Multilingual Hallucination and Omission Detection in Machine Translationcited by 35
  6. 2025Real-Time Detection of Hallucinated Entities in Long-Form Generationcited by 33
  7. 2023Teaching Language Models to Hallucinate Less with Synthetic Taskscited by 31
  8. 2024A Probabilistic Framework for LLM Hallucination Detection via Belief Tree Propagationcited by 21
  9. 2025SECOND: Mitigating Perceptual Hallucination in Vision-Language Models via Selective and Contrastive Decodingcited by 19
  10. 2025Uncertainty Quantification for Language Models: A Suite of Black-Box, White-Box, LLM Judge, and Ensemble Scorerscited by 16
  11. 2025Beyond Token Probes: Hallucination Detection via Activation Tensors with ACT-ViTcited by 16
  12. 2023The Inside Story: Towards Better Understanding of Machine Translation Neural Evaluation Metricscited by 15
  13. 2023Towards Explainable Evaluation Metrics for Machine Translationcited by 15
  14. 2024Locret: Enhancing Eviction in Long-Context LLM Inference with Trained Retaining Heads on Consumer-Grade Devicescited by 15
  15. 2025Control the Temperature: Selective Sampling for Diverse and High-Quality LLM Outputscited by 15
  16. 2022Optimal Transport for Unsupervised Hallucination Detection in Neural Machine Translationcited by 13
  17. 2025HalluEntity: Benchmarking and Understanding Entity-Level Hallucination Detectioncited by 13
  18. 2023Simple Token-Level Confidence Improves Caption Correctnesscited by 11
  19. 2025Beyond Next Token Probabilities: Learnable, Fast Detection of Hallucinations and Data Contamination on LLM Output Distributionscited by 11
  20. 2024ERBench: An Entity-Relationship based Automatically Verifiable Hallucination Benchmark for Large Language Modelscited by 9
  21. 2023Explicit Syntactic Guidance for Neural Text Generationcited by 8
  22. 2025Enhancing Reasoning to Adapt Large Language Models for Domain-Specific Applicationscited by 8
  23. 2025Beyond Decoder-only: Large Language Models Can be Good Encoders for Machine Translationcited by 8
  24. 2024A Survey of Automatic Hallucination Evaluation on Natural Language Generationcited by 7
  25. 2023On Evaluating Multilingual Compositional Generalization with Translated Datasetscited by 6
  26. 2025A Unified Definition of Hallucination: It's The World Model, Stupid!cited by 6
  27. 2025Hallucination Detection: A Probabilistic Framework Using Embeddings Distance Analysiscited by 5
  28. 2025Neural Message-Passing on Attention Graphs for Hallucination Detectioncited by 5
  29. 2024Quantifying Positional Biases in Text Embedding Modelscited by 4
  30. 2026Hear Both Sides: Efficient Multi-Agent Debate via Diversity-Aware Message Retentioncited by 4
  31. 2026Mitigating Manifold Departure: Uncertainty-Aware Subspace Rectification for Trustworthy MLLM Decodingcited by 4
  32. 2022Rare but Severe Neural Machine Translation Errors Induced by Minimal Deletion: An Empirical Study on Chinese and Englishcited by 3
  33. 2023Progressive Translation: Improving Domain Robustness of Neural Machine Translation with Intermediate Sequencescited by 3
  34. 2025MTRE: Multi-Token Reliability Estimation for Hallucination Detection in VLMscited by 3
  35. 2023Translate your gibberish: black-box adversarial attack on machine translation systemscited by 2
  36. 2024Investigating Factuality in Long-Form Text Generation: The Roles of Self-Known and Self-Unknowncited by 2
  37. 2025Shaking to Reveal: Perturbation-Based Detection of LLM Hallucinationscited by 2
  38. 2026Beyond Majority Voting: Efficient Best-Of-N with Radial Consensus Scorecited by 2

50 most cited of 60 All 60 (JSON)