mlsanityData to 7 Oct 2026

Calibration of Encoder Decoder Models for Neural Machine Translation

arXiv 1903.00802 ↗First posted 3 Mar 2019cs.LG cs.CL stat.MLcited by 67

Not on the map. Why?

Abstract (arXiv)

We study the calibration of several state of the art neural machine translation(NMT) systems built on attention-based encoder-decoder models. For structured outputs like in NMT, calibration is important not just for reliable confidence with predictions, but also for proper functioning of beam-search inference. We show that most modern NMT models are surprisingly miscalibrated even when conditioned on the true previous tokens. Our investigation leads to two main reasons -- severe miscalibration of EOS (end of sequence marker) and suppression of attention uncertainty. We design recalibration methods based on these signals and demonstrate improved accuracy, better sequence-level calibration, and more intuitive results from beam-search.

Key facts. Cited by 67 papers in the map (data through 7 Oct 2026). First posted on arXiv 3 Mar 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 12; 2022: 15; 2023: 18; 2024: 9; 2025: 10; 2026: 3.

67 papers.

Sort:

By area:

  1. 2023A Survey of Confidence Estimation and Calibration in Large Language Modelscited by 253
  2. 2022BRIO: Bringing Order to Abstractive Summarizationcited by 161
  3. 2024Kernel Language Entropy: Fine-grained Uncertainty Quantification for LLMs from Semantic Similaritiescited by 156
  4. 2025Uncertainty Quantification and Confidence Calibration in Large Language Models: A Surveycited by 93
  5. 2023Calibration in Deep Learning: A Survey of the State-of-the-Artcited by 72
  6. 2023On the Calibration of Large Language Models and Alignmentcited by 72
  7. 2021Sampling-Based Approximations to Minimum Bayes Risk Decoding for Neural Machine Translationcited by 63
  8. 2023Selectively Answering Ambiguous Questionscited by 63
  9. 2023Uncertainty in Natural Language Generation: From Theory to Applicationscited by 61
  10. 2022Looking for a Needle in a Haystack: A Comprehensive Study of Hallucinations in Neural Machine Translationcited by 60
  11. 2021Understanding the Properties of Minimum Bayes Risk Decoding in Neural Machine Translationcited by 56
  12. 2021Knowing More About Questions Can Help: Improving Calibration in Question Answeringcited by 47
38 more
  1. 2022Robust Calibration with Multi-domain Temperature Scalingcited by 39
  2. 2021Calibrated and Sharp Uncertainties in Deep Learning via Density Estimationcited by 33
  3. 2022Calibrated Interpretation: Confidence Estimation in Semantic Parsingcited by 33
  4. 2025Optimizing Temperature for Language Models with Multi-Sample Inferencecited by 33
  5. 2023Learning to Reject with a Fixed Predictor: Application to Decontextualizationcited by 32
  6. 2021Distributed NLI: Learning to Predict Human Opinion Distributions for Language Reasoningcited by 31
  7. 2022On the Calibration of Pre-trained Language Models using Mixup Guided by Area Under the Margin and Saliencycited by 31
  8. 2023A Survey on Out-of-Distribution Detection in NLPcited by 31
  9. 2021Diversifying Dialog Generation via Adaptive Label Smoothingcited by 25
  10. 2024Knowledge Entropy Decay during Language Model Pretraining Hinders New Knowledge Acquisitioncited by 15
  11. 2025MICE for CATs: Model-Internal Confidence Estimation for Calibrating Agents with Toolscited by 15
  12. 2022WR-ONE2SET: Towards Well-Calibrated Keyphrase Generationcited by 13
  13. 2021Smoothing and Shrinking the Sparse Seq2Seq Search Spacecited by 11
  14. 2022Calibration of Machine Reading Systems at Scalecited by 11
  15. 2023Simple Token-Level Confidence Improves Caption Correctnesscited by 11
  16. 2021Confidence-Aware Scheduled Sampling for Neural Machine Translationcited by 10
  17. 2021kFolden: k-Fold Ensemble for Out-Of-Distribution Detectioncited by 10
  18. 2022Jam or Cream First? Modeling Ambiguity in Neural Machine Translation with SCONEScited by 10
  19. 2021Beyond Noise: Mitigating the Impact of Fine-grained Semantic Divergences on Neural Machine Translationcited by 9
  20. 2022Uncertainty Determines the Adequacy of the Mode and the Tractability of Decoding in Sequence-to-Sequence Modelscited by 9
  21. 2022Automated Crossword Solvingcited by 9
  22. 2024Adaptive Decoding via Latent Preference Optimizationcited by 9
  23. 2023Reliable Gradient-free and Likelihood-free Prompt Tuningcited by 8
  24. 2023Uncertainty Aware Training to Improve Deep Learning Model Calibration for Classification of Cardiac MR Imagescited by 7
  25. 2024Text-to-SQL Calibration: No Need to Ask -- Just Rescale Model Probabilitiescited by 7
  26. 2024FedCal: Achieving Local and Global Calibration in Federated Learning via Aggregated Parameterized Scalercited by 6
  27. 2023Context-Aware Selective Label Smoothing for Calibrating Sequence Recognition Modelcited by 5
  28. 2025Calibrating LLMs for Text-to-SQL Parsing by Leveraging Sub-clause Frequenciescited by 5
  29. 2021Multi-Sentence Resampling: A Simple Approach to Alleviate Dataset Length Bias and Beam-Search Degradationcited by 4
  30. 2022Hard Gate Knowledge Distillation -- Leverage Calibration for Robust and Reliable Language Modelcited by 4
  31. 2022DC-MBR: Distributional Cooling for Minimum Bayesian Risk Decodingcited by 4
  32. 2024Predict the Next Word: Humans exhibit uncertainty in this task and language models _____cited by 4
  33. 2025Improving the Calibration of Confidence Scores in Text Generation Using the Output Distribution's Characteristicscited by 4
  34. 2023Who Needs Decoders? Efficient Estimation of Sequence-level Attributescited by 3
  35. 2023When to Read Documents or QA History: On Unified and Selective Open-domain QAcited by 3
  36. 2025Comparing Uncertainty Measurement and Mitigation Methods for Large Language Models: A Systematic Reviewcited by 3
  37. 2025Decoding Uncertainty: The Impact of Decoding Strategies for Uncertainty Estimation in Large Language Modelscited by 3
  38. 2021Searching for Search Errors in Neural Morphological Inflectioncited by 2

50 most cited of 67 All 67 (JSON)