# Uncertainty-Aware Balancing for Multilingual and Multi-Domain Neural Machine Translation Training

arXiv: https://arxiv.org/abs/2109.02284 · HTML: https://mlsanity.com/paper/2109.02284/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 51 references that are in the map.
- First posted on arXiv 6 Sep 2021.

## Builds on (most cited first, 28 of 28)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?](https://mlsanity.com/paper/1703.04977/) (2017)
- [A Call for Clarity in Reporting BLEU Scores](https://mlsanity.com/paper/1804.08771/) (2018)
- [fairseq: A Fast, Extensible Toolkit for Sequence Modeling](https://mlsanity.com/paper/1904.01038/) (2019)
- [Google's Multilingual Neural Machine Translation System: Enabling Zero-Shot Translation](https://mlsanity.com/paper/1611.04558/) (2016)
- [Beto, Bentz, Becas: The Surprising Cross-Lingual Effectiveness of BERT](https://mlsanity.com/paper/1904.09077/) (2019)
- [Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges](https://mlsanity.com/paper/1907.05019/) (2019)
- [Massively Multilingual Neural Machine Translation](https://mlsanity.com/paper/1903.00089/) (2019)
- [Unsupervised Quality Estimation for Neural Machine Translation](https://mlsanity.com/paper/2005.10608/) (2020)
- [Gradient Vaccine: Investigating and Improving Multi-task Optimization in Massively Multilingual Models](https://mlsanity.com/paper/2010.05874/) (2020)
- [Learning how to Active Learn: A Deep Reinforcement Learning Approach](https://mlsanity.com/paper/1708.02383/) (2017)
- [Quantifying Uncertainties in Natural Language Processing Tasks](https://mlsanity.com/paper/1811.07253/) (2018)
- [Balancing Training for Multilingual Neural Machine Translation](https://mlsanity.com/paper/2004.06748/) (2020)
- [Optimizing Data Usage via Differentiable Rewards](https://mlsanity.com/paper/1911.10088/) (2019)
- [Improving Back-Translation with Uncertainty-based Confidence Estimation](https://mlsanity.com/paper/1909.00157/) (2019)
- [Confidence Modeling for Neural Semantic Parsing](https://mlsanity.com/paper/1805.04604/) (2018)
- [Wat zei je? Detecting Out-of-Distribution Translations with Variational Transformers](https://mlsanity.com/paper/2006.08344/) (2020)
- [Multilingual Part-of-Speech Tagging with Bidirectional Long Short-Term Memory Models and Auxiliary Loss](https://mlsanity.com/paper/1604.05529/) (2016)
- [Complete Multilingual Neural Machine Translation](https://mlsanity.com/paper/2010.10239/) (2020)
- [What's in a Domain? Learning Domain-Robust Text Representations using Adversarial Training](https://mlsanity.com/paper/1805.06088/) (2018)
- [Multi-Domain Neural Machine Translation](https://mlsanity.com/paper/1805.02282/) (2018)
- [Target Conditioned Sampling: Optimizing Data Selection for Multilingual Neural Machine Translation](https://mlsanity.com/paper/1905.08212/) (2019)
- [Neural Machine Translation Training in a Multi-Domain Scenario](https://mlsanity.com/paper/1708.08712/) (2017)
- [Semi-supervised Stochastic Multi-Domain Learning using Variational Inference](https://mlsanity.com/paper/1906.02897/) (2019)

## Built on by (most cited first, 19 of 19)

- [Bactrian-X: Multilingual Replicable Instruction-Following Models with Low-Rank Adaptation](https://mlsanity.com/paper/2305.15011/) (2023)
- [(Perhaps) Beyond Human Translation: Harnessing Multi-Agent Collaboration for Translating Ultra-Long Literary Texts](https://mlsanity.com/paper/2405.11804/) (2024)
- [Uncertainty in Natural Language Processing: Sources, Quantification, and Applications](https://mlsanity.com/paper/2306.04459/) (2023)
- [A Paradigm Shift: The Future of Machine Translation Lies with Large Language Models](https://mlsanity.com/paper/2305.01181/) (2023)
- [Mixture-of-Skills: Learning to Optimize Data Usage for Fine-Tuning Large Language Models](https://mlsanity.com/paper/2406.08811/) (2024)
- [Universal Conditional Masked Language Pre-training for Neural Machine Translation](https://mlsanity.com/paper/2203.09210/) (2022)
- [Towards Higher Pareto Frontier in Multilingual Machine Translation](https://mlsanity.com/paper/2305.15718/) (2023)
- [DynamixSFT: Dynamic Mixture Optimization of Instruction Tuning Collections](https://mlsanity.com/paper/2508.12116/) (2025)
- [Importance-Aware Data Augmentation for Document-Level Neural Machine Translation](https://mlsanity.com/paper/2401.15360/) (2024)
- [CONGRAD:Conflicting Gradient Filtering for Multilingual Preference Alignment](https://mlsanity.com/paper/2503.23777/) (2025)
- [Unifying the Convergences in Multilingual Neural Machine Translation](https://mlsanity.com/paper/2205.01620/) (2022)
- [Realistic Conversational Question Answering with Answer Selection based on Calibrated Confidence and Uncertainty Measurement](https://mlsanity.com/paper/2302.05137/) (2023)
- [HBO: Hierarchical Balancing Optimization for Fine-Tuning Large Language Models](https://mlsanity.com/paper/2505.12300/) (2025)
- [NIRANTAR: Continual Learning with New Languages and Domains on Real-world Speech Data](https://mlsanity.com/paper/2507.00534/) (2025)
- [Uncertainty Quantification for Evaluating Machine Translation Bias](https://mlsanity.com/paper/2507.18338/) (2025)
- [Robust Guidance for Unsupervised Data Selection: Capturing Perplexing Named Entities for Domain-Specific Machine Translation](https://mlsanity.com/paper/2402.19267/) (2024)
- [Can the Variation of Model Weights be used as a Criterion for Self-Paced Multilingual NMT?](https://mlsanity.com/paper/2410.04147/) (2024)
- [Towards Reliable Neural Machine Translation with Consistency-Aware Meta-Learning](https://mlsanity.com/paper/2303.10966/) (2023)
- [Optimizing the Training Schedule of Multilingual NMT using Reinforcement Learning](https://mlsanity.com/paper/2410.06118/) (2024)

## Abstract (arXiv)

Learning multilingual and multi-domain translation model is challenging as the heterogeneous and imbalanced data make the model converge inconsistently over different corpora in real world. One common practice is to adjust the share of each corpus in the training, so that the learning process is balanced and low-resource cases can benefit from the high resource ones. However, automatic balancing methods usually depend on the intra- and inter-dataset characteristics, which is usually agnostic or requires human priors. In this work, we propose an approach, MultiUAT, that dynamically adjusts the training data usage based on the model's uncertainty on a small set of trusted clean data for multi-corpus machine translation. We experiments with two classes of uncertainty measures on multilingual (16 languages with 4 settings) and multi-domain settings (4 for in-domain and 2 for out-of-domain on English-German translation) and demonstrate our approach MultiUAT substantially outperforms its baselines, including both static and dynamic strategies. We analyze the cross-domain transfer and show the deficiency of static and similarity based methods.