# Findings of the First Shared Task on Machine Translation Robustness

arXiv: https://arxiv.org/abs/1906.11943 · HTML: https://mlsanity.com/paper/1906.11943/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Jun 2019.

## Built on by (most cited first, 17 of 17)

- [Robust Open-Vocabulary Translation from Visual Text Representations](https://mlsanity.com/paper/2104.08211/) (2021)
- [Pretrained Models for Multilingual Federated Learning](https://mlsanity.com/paper/2206.02291/) (2022)
- [Why don't people use character-level machine translation?](https://mlsanity.com/paper/2110.08191/) (2021)
- [FrugalScore: Learning Cheaper, Lighter and Faster Evaluation Metricsfor Automatic Text Generation](https://mlsanity.com/paper/2110.08559/) (2021)
- [JParaCrawl v3.0: A Large-scale English-Japanese Parallel Corpus](https://mlsanity.com/paper/2202.12607/) (2022)
- [Understanding Model Robustness to User-generated Noisy Texts](https://mlsanity.com/paper/2110.07428/) (2021)
- [Enhancing Translation Accuracy of Large Language Models through Continual Pre-Training on Parallel Data](https://mlsanity.com/paper/2407.03145/) (2024)
- [Addressing the Vulnerability of NMT in Input Perturbations](https://mlsanity.com/paper/2104.09810/) (2021)
- [Pushing the Right Buttons: Adversarial Evaluation of Quality Estimation](https://mlsanity.com/paper/2109.10859/) (2021)
- [SOTASTREAM: A Streaming Approach to Machine Translation Training](https://mlsanity.com/paper/2308.07489/) (2023)
- [Misspellings in Natural Language Processing: A survey](https://mlsanity.com/paper/2501.16836/) (2025)
- [Visual Cues and Error Correction for Translation Robustness](https://mlsanity.com/paper/2103.07352/) (2021)
- [Distributionally Robust Recurrent Decoders with Random Network Distillation](https://mlsanity.com/paper/2110.13229/) (2021)
- [Denoising Diffusion for 3D Hand Pose Estimation from Images](https://mlsanity.com/paper/2308.09523/) (2023)
- [Doubly-Trained Adversarial Data Augmentation for Neural Machine Translation](https://mlsanity.com/paper/2110.05691/) (2021)
- [Two Hands Are Better Than One: Resolving Hand to Hand Intersections via Occupancy Networks](https://mlsanity.com/paper/2404.05414/) (2024)
- [Pseudo-Label Training and Model Inertia in Neural Machine Translation](https://mlsanity.com/paper/2305.11808/) (2023)

## Abstract (arXiv)

We share the findings of the first shared task on improving robustness of Machine Translation (MT). The task provides a testbed representing challenges facing MT models deployed in the real world, and facilitates new approaches to improve models; robustness to noisy input and domain mismatch. We focus on two language pairs (English-French and English-Japanese), and the submitted systems are evaluated on a blind test set consisting of noisy comments on Reddit and professionally sourced translations. As a new task, we received 23 submissions by 11 participating teams from universities, companies, national labs, etc. All submitted systems achieved large improvements over baselines, with the best improvement having +22.33 BLEU. We evaluated submissions by both human judgment and automatic evaluation (BLEU), which shows high correlations (Pearson's r = 0.94 and 0.95). Furthermore, we conducted a qualitative analysis of the submitted systems using compare-mt, which revealed their salient differences in handling challenges in this task. Such analysis provides additional insights when there is occasional disagreement between human judgment and BLEU, e.g. systems better at producing colloquial expressions received higher score from human judgment.