# Understanding the Impact of UGC Specificities on Translation Quality

arXiv: https://arxiv.org/abs/2110.12551 · HTML: https://mlsanity.com/paper/2110.12551/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 25 references that are in the map.
- First posted on arXiv 24 Oct 2021.

## Builds on (most cited first, 9 of 9)

- [A Call for Clarity in Reporting BLEU Scores](https://mlsanity.com/paper/1804.08771/) (2018)
- [Synthetic and Natural Noise Both Break Neural Machine Translation](https://mlsanity.com/paper/1711.02173/) (2017)
- [Fully Character-Level Neural Machine Translation without Explicit Segmentation](https://mlsanity.com/paper/1610.03017/) (2016)
- [MTNT: A Testbed for Machine Translation of Noisy Text](https://mlsanity.com/paper/1809.00388/) (2018)
- [OpenNMT: Neural Machine Translation Toolkit](https://mlsanity.com/paper/1805.11462/) (2018)
- [XNMT: The eXtensible Neural Machine Translation Toolkit](https://mlsanity.com/paper/1803.00188/) (2018)
- [Treebanking User-Generated Content: a UD Based Overview of Guidelines, Corpora and Unified Recommendations](https://mlsanity.com/paper/2011.02063/) (2020)
- [Machine Translation of Restaurant Reviews: New Corpus for Domain Adaptation and Robustness](https://mlsanity.com/paper/1910.14589/) (2019)
- [PheMT: A Phenomenon-wise Dataset for Machine Translation Robustness on User-Generated Contents](https://mlsanity.com/paper/2011.02121/) (2020)

## Built on by (most cited first, 3 of 3)

- [Making Sentence Embeddings Robust to User-Generated Content](https://mlsanity.com/paper/2403.17220/) (2024)
- [MultiLexNorm++: A Unified Benchmark and a Generative Model for Lexical Normalization for Asian Languages](https://mlsanity.com/paper/2601.16623/) (2026)
- [When the Gold Standard Isn't Necessarily Standard: Challenges of Evaluating the Translation of User-Generated Content](https://mlsanity.com/paper/2512.17738/) (2025)

## Abstract (arXiv)

This work takes a critical look at the evaluation of user-generated content automatic translation, the well-known specificities of which raise many challenges for MT. Our analyses show that measuring the average-case performance using a standard metric on a UGC test set falls far short of giving a reliable image of the UGC translation quality. That is why we introduce a new data set for the evaluation of UGC translation in which UGC specificities have been manually annotated using a fine-grained typology. Using this data set, we conduct several experiments to measure the impact of different kinds of UGC specificities on translation quality, more precisely than previously possible.