mlsanityData to 7 Oct 2026

On NMT Search Errors and Model Errors: Cat Got Your Tongue?

arXiv 1908.10090 ↗First posted 27 Aug 2019cs.CLcited by 114

Not on the map. Why?

Abstract (arXiv)

We report on search errors and model errors in neural machine translation (NMT). We present an exact inference procedure for neural sequence models based on a combination of beam search and depth-first search. We use our exact search to find the global best model scores under a Transformer base model for the entire WMT15 English-German test set. Surprisingly, beam search fails to find these global best model scores in most cases, even with a very large beam size of 100. For more than 50% of the sentences, the model in fact assigns its global best score to the empty translation, revealing a massive failure of neural models in properly accounting for adequacy. We show by constraining search with a minimum translation length that at the root of the problem of empty translations lies an inherent bias towards shorter translations. We conclude that vanilla NMT in its current form requires just the right amount of beam search errors, which, from a modelling perspective, is a highly unsatisfactory conclusion indeed, as the model often prefers an empty translation.

Key facts. Cited by 114 papers in the map (data through 7 Oct 2026). First posted on arXiv 27 Aug 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 1; 2021: 24; 2022: 32; 2023: 22; 2024: 19; 2025: 10; 2026: 6.

114 papers.

Sort:

By area:

  1. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,582
  2. 2023Self-Evaluation Guided Beam Search for Reasoningcited by 195
  3. 2023Grammar-Constrained Decoding for Structured NLP Tasks without Finetuningcited by 186
  4. 2022BRIO: Bringing Order to Abstractive Summarizationcited by 161
  5. 2024From Decoding to Meta-Generation: Inference-time Algorithms for Large Language Modelscited by 136
  6. 2022Internet-augmented language models through few-shot prompting for open-domain question answeringcited by 131
  7. 2023End-to-End Speech Recognition: A Surveycited by 127
  8. 2023Hallucinations in Large Multilingual Translation Modelscited by 126
  9. 2022Locally Typical Samplingcited by 108
  10. 2021Survey of Low-Resource Machine Translationcited by 92
  11. 2022On Decoding Strategies for Neural Text Generatorscited by 89
  12. 2022Coder Reviewer Reranking for Code Generationcited by 83
38 more
  1. 2021Controlled Text Generation as Continuous Optimization with Multiple Constraintscited by 70
  2. 2020Text Generation by Learning from Demonstrationscited by 64
  3. 2021Sampling-Based Approximations to Minimum Bayes Risk Decoding for Neural Machine Translationcited by 63
  4. 2023Uncertainty in Natural Language Generation: From Theory to Applicationscited by 61
  5. 2022Looking for a Needle in a Haystack: A Comprehensive Study of Hallucinations in Neural Machine Translationcited by 60
  6. 2021Understanding the Properties of Minimum Bayes Risk Decoding in Neural Machine Translationcited by 56
  7. 2021A Systematic Investigation of Commonsense Knowledge in Large Language Modelscited by 56
  8. 2021Machine Translation Decoding beyond Beam Searchcited by 55
  9. 2022LENS: A Learnable Evaluation Metric for Text Simplificationcited by 54
  10. 2022Quality-Aware Decoding for Neural Machine Translationcited by 49
  11. 2023It's MBR All the Way Down: Modern Generation Techniques Through the Lens of Minimum Bayes Riskcited by 49
  12. 2021Domain Adaptation and Multi-Domain Adaptation for Neural Machine Translation: A Surveycited by 46
  13. 2025The Majority is not always right: RL training for solution aggregationcited by 46
  14. 2023Tuning computer vision models with task rewardscited by 36
  15. 2023HalOmi: A Manually Annotated Benchmark for Multilingual Hallucination and Omission Detection in Machine Translationcited by 35
  16. 2023Faithfulness-Aware Decoding Strategies for Abstractive Summarizationcited by 31
  17. 2023Closing the Curious Case of Neural Text Degenerationcited by 30
  18. 2022Reward Gaming in Conditional Text Generationcited by 29
  19. 2023Why do Nearest Neighbor Language Models Work?cited by 29
  20. 2021First the worst: Finding better gender translations during beam searchcited by 28
  21. 2021Why don't people use character-level machine translation?cited by 27
  22. 2022Examining Scaling and Transfer of Language Model Architectures for Machine Translationcited by 26
  23. 2024Regularized Best-of-N Sampling with Minimum Bayes Risk Objective for Language Model Alignmentcited by 24
  24. 2025Fast Controlled Generation from Language Models with Adaptive Weighted Rejection Samplingcited by 20
  25. 2021Straight to the Gradient: Learning to Use Novel Tokens for Neural Text Generationcited by 19
  26. 2023Generative Models as a Complex Systems Science: How can we make sense of large language model behavior?cited by 19
  27. 2024Planning Ahead in Generative Retrieval: Guiding Autoregressive Generation through Simultaneous Decodingcited by 19
  28. 2021Determinantal Beam Searchcited by 18
  29. 2022Evaluating Distributional Distortion in Neural Language Modelingcited by 18
  30. 2022On the probability-quality paradox in language generationcited by 16
  31. 2023MixCE: Training Autoregressive Language Models by Mixing Forward and Reverse Cross-Entropiescited by 16
  32. 2024QUEST: Quality-Aware Metropolis-Hastings Sampling for Machine Translationcited by 16
  33. 2025Causal Estimation of Tokenisation Biascited by 16
  34. 2024Hyperparameter-Free Approach for Faster Minimum Bayes Risk Decodingcited by 15
  35. 2024Exact Byte-Level Probabilities from Tokenized Language Models for FIM-Tasks and Model Ensemblescited by 15
  36. 2022Viterbi Decoding of Directed Acyclic Transformer for Non-Autoregressive Machine Translationcited by 13
  37. 2024Don't Rank, Combine! Combining Machine Translation Hypotheses Using Quality Estimationcited by 13
  38. 2025RESTRAIN: From Spurious Votes to Signals -- Self-Driven RL with Self-Penalizationcited by 13

50 most cited of 114 All 114 (JSON)