mlsanityData to 14 Oct 2026

Prevent the Language Model from being Overconfident in Neural Machine Translation

arXiv 2105.11098 ↗First posted 24 May 2021cs.CL cs.AIcited by 34

Not on the map. Why?

Abstract (arXiv)

The Neural Machine Translation (NMT) model is essentially a joint language model conditioned on both the source sentence and partial translation. Therefore, the NMT model naturally involves the mechanism of the Language Model (LM) that predicts the next token only based on partial translation. Despite its success, NMT still suffers from the hallucination problem, generating fluent but inadequate translations. The main reason is that NMT pays excessive attention to the partial translation while neglecting the source sentence to some extent, namely overconfidence of the LM. Accordingly, we define the Margin between the NMT and the LM, calculated by subtracting the predicted probability of the LM from that of the NMT model for each token. The Margin is negatively correlated to the overconfidence degree of the LM. Based on the property, we propose a Margin-based Token-level Objective (MTO) and a Margin-based Sentencelevel Objective (MSO) to maximize the Margin for preventing the LM from being overconfident. Experiments on WMT14 English-to-German, WMT19 Chinese-to-English, and WMT14 English-to-French translation tasks demonstrate the effectiveness of our approach, with 1.36, 1.50, and 0.63 BLEU improvements, respectively, compared to the Transformer baseline. The human evaluation further verifies that our approaches improve translation adequacy as well as fluency.

Key facts. Cited by 34 papers in the map (data through 14 Oct 2026). Builds on 39 of its 46 references that are in the map. First posted on arXiv 24 May 2021.

Builds on

39 of 46 references.

References · 39

  1. 2017Attention Is All You Needcited by 60,361
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,548
  4. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,057
  5. 2014Sequence to Sequence Learning with Neural Networkscited by 2,986
  6. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
  7. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,488
  8. 2015Effective Approaches to Attention-based Neural Machine Translationcited by 1,026
  9. 2017OpenNMT: Open-source Toolkit for Neural Machine Translationcited by 251
  10. 2018Contextual Augmentation: Data Augmentation by Words with Paradigmatic Relationscited by 201
  11. 2015On Using Monolingual Corpora in Neural Machine Translationcited by 150
  12. 2020On Exposure Bias, Hallucination and Domain Shift in Neural Machine Translationcited by 127
27 more
  1. 2019Bridging the Gap between Training and Inference for Neural Machine Translationcited by 123
  2. 2015Minimum Risk Training for Neural Machine Translationcited by 115
  3. 2016Modeling Coverage for Neural Machine Translationcited by 110
  4. 2018Conditional BERT Contextual Augmentationcited by 107
  5. 2017Cold Fusion: Training Seq2Seq Models Together with Language Modelscited by 88
  6. 2019Domain Robustness in Neural Machine Translationcited by 65
  7. 2019Curriculum Learning for Domain Adaptation in Neural Machine Translationcited by 59
  8. 2019Soft Contextual Data Augmentation for Neural Machine Translationcited by 49
  9. 2018Modeling Localness for Self-Attention Networkscited by 43
  10. 2018Dual Conditional Cross-Entropy Filtering of Noisy Parallel Corporacited by 40
  11. 2020Language Model Prior for Low-Resource Neural Machine Translationcited by 37
  12. 2019Improving Back-Translation with Uncertainty-based Confidence Estimationcited by 34
  13. 2017Dynamic Data Selection for Neural Machine Translationcited by 33
  14. 2018Simple Fusion: Return of the Language Modelcited by 30
  15. 2016Neural Machine Translation with Reconstructioncited by 28
  16. 2019Context-Aware Self-Attention Networkscited by 25
  17. 2019On the use of BERT for Neural Machine Translationcited by 23
  18. 2019Dynamically Composing Domain-Data Selection with Clean-Data Selection by "Co-Curricular Learning" for Neural Machine Translationcited by 20
  19. 2020Parallel Corpus Filtering via Pre-trained Language Modelscited by 15
  20. 2018DTMT: A Novel Deep Transition Architecture for Neural Machine Translationcited by 13
  21. 2019Modeling Fluency and Faithfulness for Diverse Neural Machine Translationcited by 13
  22. 2020Multi-Unit Transformers for Neural Machine Translationcited by 13
  23. 2018Neural Machine Translation with Adequacy-Oriented Learningcited by 8
  24. 2020GRET: Global Representation Enhanced Transformercited by 3
  25. 2016Interactive Attention for Neural Machine Translationcited by 2
  26. 2018Neural Machine Translation with Key-Value Memory-Augmented Attentioncited by 2
  27. 2020Dual Past and Future for Neural Machine Translationcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 1; 2022: 7; 2023: 9; 2024: 12; 2025: 4; 2026: 1.

34 papers.

Sort:

By area:

  1. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,601
  2. 2023Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models' Alignmentcited by 401
  3. 2023A Survey on Multimodal Large Language Models for Autonomous Drivingcited by 290
  4. 2023Conformal Language Modelingcited by 170
  5. 2024API Is Enough: Conformal Prediction for Large Language Models Without Logit-Accesscited by 69
  6. 2024SpecExec: Massively Parallel Speculative Decoding for Interactive LLM Inference on Consumer Devicescited by 46
  7. 2024Distract Large Language Models for Automatic Jailbreak Attackcited by 45
  8. 2024AI Hallucinations: A Misnomer Worth Clarifyingcited by 34
  9. 2025MedHallu: A Comprehensive Benchmark for Detecting Medical Hallucinations in Large Language Modelscited by 34
  10. 2022Towards Improving Faithfulness in Abstractive Summarizationcited by 27
  11. 2023Zero-shot Faithfulness Evaluation for Text Summarization with Foundation Language Modelcited by 21
  12. 2024Hallucination Detection and Hallucination Mitigation: An Investigationcited by 21
22 more
  1. 2022On the Complementarity between Pre-Training and Random-Initialization for Resource-Rich Machine Translationcited by 15
  2. 2022Improving Simultaneous Machine Translation with Monolingual Datacited by 13
  3. 2024ElecBench: a Power Dispatch Evaluation Benchmark for Large Language Modelscited by 13
  4. 2021Dissecting Generation Modes for Abstractive Summarization Models via Ablation and Attributioncited by 10
  5. 2023Robust Natural Language Understanding with Residual Attention Debiasingcited by 10
  6. 2025Smoothing Out Hallucinations: Mitigating LLM Hallucination with Smoothed Knowledge Distillationcited by 10
  7. 2023Self-Evolution Learning for Discriminative Language Model Pretrainingcited by 8
  8. 2022Confidence Based Bidirectional Global Context Aware Training Framework for Neural Machine Translationcited by 7
  9. 2024Flexible and Adaptable Summarization via Expertise Separationcited by 7
  10. 2024Beyond Persuasion: Towards Conversational Recommender System with Credible Explanationscited by 7
  11. 2022Conditional Bilingual Mutual Information Based Adaptive Training for Neural Machine Translationcited by 6
  12. 2022HaRiM^+: Evaluating Summary Quality with Hallucination Riskcited by 6
  13. 2025Length-Controlled Margin-Based Preference Optimization without Reference Modelcited by 6
  14. 2024WSC+: Enhancing The Winograd Schema Challenge Using Tree-of-Expertscited by 5
  15. 2023Revealing User Familiarity Bias in Task-Oriented Dialogue via Interactive Evaluationcited by 4
  16. 2025Comparing Uncertainty Measurement and Mitigation Methods for Large Language Models: A Systematic Reviewcited by 4
  17. 2023Progressive Translation: Improving Domain Robustness of Neural Machine Translation with Intermediate Sequencescited by 3
  18. 2024Guiding Reinforcement Learning Using Uncertainty-Aware Large Language Modelscited by 3
  19. 2024On the Hallucination in Simultaneous Machine Translationcited by 1
  20. 2022Lack of Fluency is Hurting Your Translation Modelcited by 0
  21. 2024Non-Fluent Synthetic Target-Language Data Improve Neural Machine Translationcited by 0
  22. 2026Hallucinations in LLMs: A Lifecycle-Based Survey of Causes, Detection, Mitigation, and Preventioncited by 0