# MTNT: A Testbed for Machine Translation of Noisy Text

arXiv: https://arxiv.org/abs/1809.00388 · HTML: https://mlsanity.com/paper/1809.00388/

## Key facts

- Cited by 54 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Sep 2018.

## Built on by (most cited first, 30 of 54)

- [Out-of-Distribution Detection and Selective Generation for Conditional Language Models](https://mlsanity.com/paper/2209.15558/) (2022)
- [Is ChatGPT A Good Translator? Yes With GPT-4 As The Engine](https://mlsanity.com/paper/2301.08745/) (2023)
- [Shifts: A Dataset of Real Distributional Shift Across Multiple Large-Scale Tasks](https://mlsanity.com/paper/2107.07455/) (2021)
- [State-of-the-art generalisation research in NLP: A taxonomy and review](https://mlsanity.com/paper/2210.03050/) (2022)
- [Robust Open-Vocabulary Translation from Visual Text Representations](https://mlsanity.com/paper/2104.08211/) (2021)
- [On Compositional Generalization of Neural Machine Translation](https://mlsanity.com/paper/2105.14802/) (2021)
- [Pretrained Models for Multilingual Federated Learning](https://mlsanity.com/paper/2206.02291/) (2022)
- [CoCoA-MT: A Dataset and Benchmark for Contrastive Controlled MT with Application to Formality](https://mlsanity.com/paper/2205.04022/) (2022)
- [Shifts 2.0: Extending The Dataset of Real Distributional Shifts](https://mlsanity.com/paper/2206.15407/) (2022)
- [Distributionally Robust Models with Parametric Likelihood Ratios](https://mlsanity.com/paper/2204.06340/) (2022)
- [In-context Learning as Maintaining Coherency: A Study of On-the-fly Machine Translation Using Large Language Models](https://mlsanity.com/paper/2305.03573/) (2023)
- [CODET: A Benchmark for Contrastive Dialectal Evaluation of Machine Translation](https://mlsanity.com/paper/2305.17267/) (2023)
- [Did Translation Models Get More Robust Without Anyone Even Noticing?](https://mlsanity.com/paper/2403.03923/) (2024)
- [As Easy as 1, 2, 3: Behavioural Testing of NMT Systems for Numerical Translation](https://mlsanity.com/paper/2107.08357/) (2021)
- [Understanding Model Robustness to User-generated Noisy Texts](https://mlsanity.com/paper/2110.07428/) (2021)
- [All You Need In Sign Language Production](https://mlsanity.com/paper/2201.01609/) (2022)
- [Addressing the Vulnerability of NMT in Input Perturbations](https://mlsanity.com/paper/2104.09810/) (2021)
- [BranchNorm: Robustly Scaling Extremely Deep Transformers](https://mlsanity.com/paper/2305.02790/) (2023)
- [Noisy UGC Translation at the Character Level: Revisiting Open-Vocabulary Capabilities and Robustness of Char-Based Models](https://mlsanity.com/paper/2110.12552/) (2021)
- [Logit-Based Ensemble Distribution Distillation for Robust Autoregressive Sequence Uncertainties](https://mlsanity.com/paper/2305.10384/) (2023)
- [Break it, Imitate it, Fix it: Robustness by Generating Human-Like Attacks](https://mlsanity.com/paper/2310.16955/) (2023)
- [MOLE: Metadata Extraction and Validation in Scientific Papers Using LLMs](https://mlsanity.com/paper/2505.19800/) (2025)
- [Monotonic Simultaneous Translation with Chunk-wise Reordering and Refinement](https://mlsanity.com/paper/2110.09646/) (2021)
- [Morphological Processing of Low-Resource Languages: Where We Are and What's Next](https://mlsanity.com/paper/2203.08909/) (2022)
- [Generating Authentic Adversarial Examples beyond Meaning-preserving with Doubly Round-trip Translation](https://mlsanity.com/paper/2204.08689/) (2022)
- [Understanding the Impact of UGC Specificities on Translation Quality](https://mlsanity.com/paper/2110.12551/) (2021)
- [GLAMI-1M: A Multilingual Image-Text Fashion Dataset](https://mlsanity.com/paper/2211.14451/) (2022)
- [An Investigation of Noise in Morphological Inflection](https://mlsanity.com/paper/2305.16581/) (2023)
- [Ask Language Model to Clean Your Noisy Translation Data](https://mlsanity.com/paper/2310.13469/) (2023)
- [Misspellings in Natural Language Processing: A survey](https://mlsanity.com/paper/2501.16836/) (2025)

## Abstract (arXiv)

Noisy or non-standard input text can cause disastrous mistranslations in most modern Machine Translation (MT) systems, and there has been growing research interest in creating noise-robust MT systems. However, as of yet there are no publicly available parallel corpora of with naturally occurring noisy inputs and translations, and thus previous work has resorted to evaluating on synthetically created datasets. In this paper, we propose a benchmark dataset for Machine Translation of Noisy Text (MTNT), consisting of noisy comments on Reddit (www.reddit.com) and professionally sourced translations. We commissioned translations of English comments into French and Japanese, as well as French and Japanese comments into English, on the order of 7k-37k sentences per language pair. We qualitatively and quantitatively examine the types of noise included in this dataset, then demonstrate that existing MT models fail badly on a number of noise-related phenomena, even after performing adaptation on a small training set of in-domain data. This indicates that this dataset can provide an attractive testbed for methods tailored to handling noisy text in MT. The data is publicly available at www.cs.cmu.edu/~pmichel1/mtnt/.