# Visual Cues and Error Correction for Translation Robustness

arXiv: https://arxiv.org/abs/2103.07352 · HTML: https://mlsanity.com/paper/2103.07352/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 56 references that are in the map.
- First posted on arXiv 12 Mar 2021.

## Builds on (most cited first, 30 of 33)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering](https://mlsanity.com/paper/1707.07998/) (2017)
- [Beyond Accuracy: Behavioral Testing of NLP models with CheckList](https://mlsanity.com/paper/2005.04118/) (2020)
- [Synthetic and Natural Noise Both Break Neural Machine Translation](https://mlsanity.com/paper/1711.02173/) (2017)
- [Multi30K: Multilingual English-German Image Descriptions](https://mlsanity.com/paper/1605.00459/) (2016)
- [Transformers without Tears: Improving the Normalization of Self-Attention](https://mlsanity.com/paper/1910.05895/) (2019)
- [Robust Neural Machine Translation with Doubly Adversarial Inputs](https://mlsanity.com/paper/1906.02443/) (2019)
- [On Adversarial Examples for Character-Level Neural Machine Translation](https://mlsanity.com/paper/1806.09030/) (2018)
- [Findings of the Second Shared Task on Multimodal Machine Translation and Multilingual Image Description](https://mlsanity.com/paper/1710.07177/) (2017)
- [AdvAug: Robust Adversarial Augmentation for Neural Machine Translation](https://mlsanity.com/paper/2006.11834/) (2020)
- [MTNT: A Testbed for Machine Translation of Noisy Text](https://mlsanity.com/paper/1809.00388/) (2018)
- [Probing the Need for Visual Context in Multimodal Machine Translation](https://mlsanity.com/paper/1903.08678/) (2019)
- [Training on Synthetic Noise Improves Robustness to Natural Noise in Machine Translation](https://mlsanity.com/paper/1902.01509/) (2019)
- [Doubly-Attentive Decoder for Multi-modal Neural Machine Translation](https://mlsanity.com/paper/1702.01287/) (2017)
- [Incorporating Global Visual Features into Attention-Based Neural Machine Translation](https://mlsanity.com/paper/1701.06521/) (2017)
- [Improving Robustness of Machine Translation with Synthetic Noise](https://mlsanity.com/paper/1902.09508/) (2019)
- [Towards Robust Neural Machine Translation](https://mlsanity.com/paper/1805.06130/) (2018)
- [Attention Strategies for Multi-Source Sequence-to-Sequence Learning](https://mlsanity.com/paper/1704.06567/) (2017)
- [Distilling Translations with Visual Awareness](https://mlsanity.com/paper/1906.07701/) (2019)
- [Multimodal Machine Translation through Visuals and Speech](https://mlsanity.com/paper/1911.12798/) (2019)
- [Naver Labs Europe's Systems for the WMT19 Machine Translation Robustness Task](https://mlsanity.com/paper/1907.06488/) (2019)
- [Input Combination Strategies for Multi-Source Transformer Decoder](https://mlsanity.com/paper/1811.04716/) (2018)
- [Multimodal Attention for Neural Machine Translation](https://mlsanity.com/paper/1609.03976/) (2016)
- [Findings of the First Shared Task on Machine Translation Robustness](https://mlsanity.com/paper/1906.11943/) (2019)
- [Simultaneous Machine Translation with Visual Context](https://mlsanity.com/paper/2009.07310/) (2020)
- [Improving Neural Machine Translation Robustness via Data Augmentation: Beyond Back Translation](https://mlsanity.com/paper/1910.03009/) (2019)
- [Neural Machine Translation of Text from Non-Native Speakers](https://mlsanity.com/paper/1808.06267/) (2018)
- [Towards Multimodal Simultaneous Neural Machine Translation](https://mlsanity.com/paper/2004.03180/) (2020)

## Built on by (most cited first, 2 of 2)

- [Misspellings in Natural Language Processing: A survey](https://mlsanity.com/paper/2501.16836/) (2025)
- [Vision-Grounded Machine Interpreting: Improving the Translation Process through Visual Cues](https://mlsanity.com/paper/2509.23957/) (2025)

## Abstract (arXiv)

Neural Machine Translation models are sensitive to noise in the input texts, such as misspelled words and ungrammatical constructions. Existing robustness techniques generally fail when faced with unseen types of noise and their performance degrades on clean texts. In this paper, we focus on three types of realistic noise that are commonly generated by humans and introduce the idea of visual context to improve translation robustness for noisy texts. In addition, we describe a novel error correction training regime that can be used as an auxiliary task to further improve translation robustness. Experiments on English-French and English-German translation show that both multimodal and error correction components improve model robustness to noisy texts, while still retaining translation quality on clean texts.