# Evaluating Multiway Multilingual NMT in the Turkic Languages

arXiv: https://arxiv.org/abs/2109.06262 · HTML: https://mlsanity.com/paper/2109.06262/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 62 references that are in the map.
- First posted on arXiv 13 Sep 2021.

## Builds on (most cited first, 20 of 20)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [fairseq: A Fast, Extensible Toolkit for Sequence Modeling](https://mlsanity.com/paper/1904.01038/) (2019)
- [Beyond English-Centric Multilingual Machine Translation](https://mlsanity.com/paper/2010.11125/) (2020)
- [The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translation](https://mlsanity.com/paper/2106.03193/) (2021)
- [Google's Multilingual Neural Machine Translation System: Enabling Zero-Shot Translation](https://mlsanity.com/paper/1611.04558/) (2016)
- [Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges](https://mlsanity.com/paper/1907.05019/) (2019)
- [CCMatrix: Mining Billions of High-Quality Parallel Sentences on the WEB](https://mlsanity.com/paper/1911.04944/) (2019)
- [Participatory Research for Low-resourced Machine Translation: A Case Study in African Languages](https://mlsanity.com/paper/2010.02353/) (2020)
- [CCAligned: A Massive Collection of Cross-Lingual Web-Document Pairs](https://mlsanity.com/paper/1911.06154/) (2019)
- [The Tatoeba Translation Challenge -- Realistic Data Sets for Low Resource and Multilingual MT](https://mlsanity.com/paper/2010.06354/) (2020)
- [Toward Multilingual Neural Machine Translation with Universal Encoder and Decoder](https://mlsanity.com/paper/1611.04798/) (2016)
- [Scientific Credibility of Machine Translation Research: A Meta-Evaluation of 769 Papers](https://mlsanity.com/paper/2106.15195/) (2021)
- [Joey NMT: A Minimalist NMT Toolkit for Novices](https://mlsanity.com/paper/1907.12484/) (2019)
- [Complete Multilingual Neural Machine Translation](https://mlsanity.com/paper/2010.10239/) (2020)
- [Linguistically Motivated Vocabulary Reduction for Neural Machine Translation from Turkish to English](https://mlsanity.com/paper/1707.09879/) (2017)
- [A Large-Scale Study of Machine Translation in the Turkic Languages](https://mlsanity.com/paper/2109.04593/) (2021)
- [Unsung Challenges of Building and Deploying Language Technologies for Low Resource Language Communities](https://mlsanity.com/paper/1912.03457/) (2019)
- [Application of Low-resource Machine Translation Techniques to Russian-Tatar Language Pair](https://mlsanity.com/paper/1910.00368/) (2019)

## Built on by (most cited first, 8 of 8)

- [Building Machine Translation Systems for the Next Thousand Languages](https://mlsanity.com/paper/2205.03983/) (2022)
- [SMOL: Professionally translated parallel data for 115 under-represented languages](https://mlsanity.com/paper/2502.12301/) (2025)
- [What changes when you randomly choose BPE merge operations? Not much](https://mlsanity.com/paper/2305.03029/) (2023)
- [KyrgyzNLP: Challenges, Progress, and Future](https://mlsanity.com/paper/2411.05503/) (2024)
- [Benchmarking Multilabel Topic Classification in the Kyrgyz Language](https://mlsanity.com/paper/2308.15952/) (2023)
- [TurkicNLP: An NLP Toolkit for Turkic Languages](https://mlsanity.com/paper/2602.19174/) (2026)
- [JoeyS2T: Minimalistic Speech-to-Text Modeling with JoeyNMT](https://mlsanity.com/paper/2210.02545/) (2022)
- [KyrgyzLLM-Bench: Benchmarking Kyrgyz Language Understanding](https://mlsanity.com/paper/2607.17173/) (2026)

## Abstract (arXiv)

Despite the increasing number of large and comprehensive machine translation (MT) systems, evaluation of these methods in various languages has been restrained by the lack of high-quality parallel corpora as well as engagement with the people that speak these languages. In this study, we present an evaluation of state-of-the-art approaches to training and evaluating MT systems in 22 languages from the Turkic language family, most of which being extremely under-explored. First, we adopt the TIL Corpus with a few key improvements to the training and the evaluation sets. Then, we train 26 bilingual baselines as well as a multi-way neural MT (MNMT) model using the corpus and perform an extensive analysis using automatic metrics as well as human evaluations. We find that the MNMT model outperforms almost all bilingual baselines in the out-of-domain test sets and finetuning the model on a downstream task of a single pair also results in a huge performance boost in both low- and high-resource scenarios. Our attentive analysis of evaluation criteria for MT models in Turkic languages also points to the necessity for further research in this direction. We release the corpus splits, test sets as well as models to the public.