# On the Inference Calibration of Neural Machine Translation

arXiv: https://arxiv.org/abs/2005.00963 · HTML: https://mlsanity.com/paper/2005.00963/

## Key facts

- Cited by 61 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 May 2020.

## Built on by (most cited first, 30 of 61)

- [A Survey of Confidence Estimation and Calibration in Large Language Models](https://mlsanity.com/paper/2311.08298/) (2023)
- [Word Alignment by Fine-tuning Embeddings on Parallel Corpora](https://mlsanity.com/paper/2101.08231/) (2021)
- [Don't Hallucinate, Abstain: Identifying LLM Knowledge Gaps via Multi-LLM Collaboration](https://mlsanity.com/paper/2402.00367/) (2024)
- [BRIO: Bringing Order to Abstractive Summarization](https://mlsanity.com/paper/2203.16804/) (2022)
- [Uncertainty Quantification and Confidence Calibration in Large Language Models: A Survey](https://mlsanity.com/paper/2503.15850/) (2025)
- [Navigating the Grey Area: How Expressions of Uncertainty and Overconfidence Affect Language Models](https://mlsanity.com/paper/2302.13439/) (2023)
- [Understanding and Detecting Hallucinations in Neural Machine Translation via Model Introspection](https://mlsanity.com/paper/2301.07779/) (2023)
- [Uncertainty in Natural Language Processing: Sources, Quantification, and Applications](https://mlsanity.com/paper/2306.04459/) (2023)
- [On Compositional Generalization of Neural Machine Translation](https://mlsanity.com/paper/2105.14802/) (2021)
- [Calibrated Interpretation: Confidence Estimation in Semantic Parsing](https://mlsanity.com/paper/2211.07443/) (2022)
- [Distributed NLI: Learning to Predict Human Opinion Distributions for Language Reasoning](https://mlsanity.com/paper/2104.08676/) (2021)
- [On the Calibration of Pre-trained Language Models using Mixup Guided by Area Under the Margin and Saliency](https://mlsanity.com/paper/2203.07559/) (2022)
- [Towards More Fine-grained and Reliable NLP Performance Prediction](https://mlsanity.com/paper/2102.05486/) (2021)
- [Exploring Predictive Uncertainty and Calibration in NLP: A Study on the Impact of Method & Data Scarcity](https://mlsanity.com/paper/2210.15452/) (2022)
- [On Uncertainty Calibration and Selective Generation in Probabilistic Neural Summarization: A Benchmark Study](https://mlsanity.com/paper/2304.08653/) (2023)
- [Self-Training Sampling with Monolingual Data Uncertainty for Neural Machine Translation](https://mlsanity.com/paper/2106.00941/) (2021)
- [Language Models are Good Translators](https://mlsanity.com/paper/2106.13627/) (2021)
- [LACIE: Listener-Aware Finetuning for Confidence Calibration in Large Language Models](https://mlsanity.com/paper/2405.21028/) (2024)
- [MICE for CATs: Model-Internal Confidence Estimation for Calibrating Agents with Tools](https://mlsanity.com/paper/2504.20168/) (2025)
- [Bridging Subword Gaps in Pretrain-Finetune Paradigm for Natural Language Generation](https://mlsanity.com/paper/2106.06125/) (2021)
- [Teaching LLMs to Abstain across Languages via Multilingual Feedback](https://mlsanity.com/paper/2406.15948/) (2024)
- [How Does Distilled Data Complexity Impact the Quality and Confidence of Non-Autoregressive Machine Translation?](https://mlsanity.com/paper/2105.12900/) (2021)
- [Adaptive Label Smoothing To Regularize Large-Scale Graph Training](https://mlsanity.com/paper/2108.13555/) (2021)
- [Calibration of Large Language Models on Code Summarization](https://mlsanity.com/paper/2404.19318/) (2024)
- [Smoothing and Shrinking the Sparse Seq2Seq Search Space](https://mlsanity.com/paper/2103.10291/) (2021)
- [WeChat Neural Machine Translation Systems for WMT21](https://mlsanity.com/paper/2108.02401/) (2021)
- [Poor Man's Quality Estimation: Predicting Reference-Based MT Metrics Without the Reference](https://mlsanity.com/paper/2301.09008/) (2023)
- [Simple Token-Level Confidence Improves Caption Correctness](https://mlsanity.com/paper/2305.07021/) (2023)
- [Unveiling Over-Memorization in Finetuning LLMs for Reasoning Tasks](https://mlsanity.com/paper/2508.04117/) (2025)
- [Confidence-Aware Scheduled Sampling for Neural Machine Translation](https://mlsanity.com/paper/2107.10427/) (2021)

## Abstract (arXiv)

Confidence calibration, which aims to make model predictions equal to the true correctness measures, is important for neural machine translation (NMT) because it is able to offer useful indicators of translation errors in the generated output. While prior studies have shown that NMT models trained with label smoothing are well-calibrated on the ground-truth training data, we find that miscalibration still remains a severe challenge for NMT during inference due to the discrepancy between training and inference. By carefully designing experiments on three language pairs, our work provides in-depth analyses of the correlation between calibration and translation performance as well as linguistic properties of miscalibration and reports a number of interesting findings that might help humans better analyze, understand and improve NMT models. Based on these observations, we further propose a new graduated label smoothing method that can improve both inference calibration and translation performance.