# Universal Neural Machine Translation for Extremely Low Resource Languages

arXiv: https://arxiv.org/abs/1802.05368 · HTML: https://mlsanity.com/paper/1802.05368/

## Key facts

- Cited by 66 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Feb 2018.

## Built on by (most cited first, 30 of 66)

- [Neural Machine Translation for Low-Resource Languages: A Survey](https://mlsanity.com/paper/2106.15115/) (2021)
- [Beyond Distillation: Task-level Mixture-of-Experts for Efficient Inference](https://mlsanity.com/paper/2110.03742/) (2021)
- [Towards Fair Federated Learning with Zero-Shot Data Augmentation](https://mlsanity.com/paper/2104.13417/) (2021)
- [Domain Adaptation and Multi-Domain Adaptation for Neural Machine Translation: A Survey](https://mlsanity.com/paper/2104.06951/) (2021)
- [Counter-Interference Adapter for Multilingual Machine Translation](https://mlsanity.com/paper/2104.08154/) (2021)
- [Towards Continual Learning for Multilingual Machine Translation via Vocabulary Substitution](https://mlsanity.com/paper/2103.06799/) (2021)
- [Continual Mixed-Language Pre-Training for Extremely Low-Resource Neural Machine Translation](https://mlsanity.com/paper/2105.03953/) (2021)
- [Language Tags Matter for Zero-Shot Neural Machine Translation](https://mlsanity.com/paper/2106.07930/) (2021)
- [Importance-based Neuron Allocation for Multilingual Neural Machine Translation](https://mlsanity.com/paper/2107.06569/) (2021)
- [Lego-MT: Learning Detachable Models for Massively Multilingual Machine Translation](https://mlsanity.com/paper/2212.10551/) (2022)
- [Think about it! Improving defeasible reasoning by first modeling the question scenario](https://mlsanity.com/paper/2110.12349/) (2021)
- [ProxyLM: Predicting Language Model Performance on Multilingual Tasks via Proxy Models](https://mlsanity.com/paper/2406.09334/) (2024)
- [Text is Text, No Matter What: Unifying Text Recognition using Knowledge Distillation](https://mlsanity.com/paper/2107.12087/) (2021)
- [A Survey on Low-Resource Neural Machine Translation](https://mlsanity.com/paper/2107.04239/) (2021)
- [Endangered Languages are not Low-Resourced!](https://mlsanity.com/paper/2103.09567/) (2021)
- [Optimizing Language Augmentation for Multilingual Large Language Models: A Case Study on Korean](https://mlsanity.com/paper/2403.10882/) (2024)
- [How Vocabulary Sharing Facilitates Multilingualism in LLaMA?](https://mlsanity.com/paper/2311.09071/) (2023)
- [Small Languages, Big Models: A Study of Continual Training on Languages of Norway](https://mlsanity.com/paper/2412.06484/) (2024)
- [Improving Zero-shot Multilingual Neural Machine Translation by Leveraging Cross-lingual Consistency Regularization](https://mlsanity.com/paper/2305.07310/) (2023)
- [Breaking Down Multilingual Machine Translation](https://mlsanity.com/paper/2110.08130/) (2021)
- [Bridging the Linguistic Divide: A Survey on Leveraging Large Language Models for Machine Translation](https://mlsanity.com/paper/2504.01919/) (2025)
- [Low-Resource Neural Machine Translation for Southern African Languages](https://mlsanity.com/paper/2104.00366/) (2021)
- [Extremely low-resource machine translation for closely related languages](https://mlsanity.com/paper/2105.13065/) (2021)
- [Syntax and Domain Aware Model for Unsupervised Program Translation](https://mlsanity.com/paper/2302.03908/) (2023)
- [Crowdsourced Phrase-Based Tokenization for Low-Resourced Neural Machine Translation: The Case of Fon Language](https://mlsanity.com/paper/2103.08052/) (2021)
- [When Word Embeddings Become Endangered](https://mlsanity.com/paper/2103.13275/) (2021)
- [CAMERO: Consistency Regularized Ensemble of Perturbed Language Models with Weight Sharing](https://mlsanity.com/paper/2204.06625/) (2022)
- [m3P: Towards Multimodal Multilingual Translation with Multimodal Prompt](https://mlsanity.com/paper/2403.17556/) (2024)
- [Pay Better Attention to Attention: Head Selection in Multilingual and Multi-Domain Sequence Modeling](https://mlsanity.com/paper/2106.10840/) (2021)
- [MTet: Multi-domain Translation for English and Vietnamese](https://mlsanity.com/paper/2210.05610/) (2022)

## Abstract (arXiv)

In this paper, we propose a new universal machine translation approach focusing on languages with a limited amount of parallel data. Our proposed approach utilizes a transfer-learning approach to share lexical and sentence level representations across multiple source languages into one target language. The lexical part is shared through a Universal Lexical Representation to support multilingual word-level sharing. The sentence-level sharing is represented by a model of experts from all source languages that share the source encoders with all other languages. This enables the low-resource language to utilize the lexical and sentence representations of the higher resource languages. Our approach is able to achieve 23 BLEU on Romanian-English WMT2016 using a tiny parallel corpus of 6k sentences, compared to the 18 BLEU of strong baseline system which uses multilingual training and back-translation. Furthermore, we show that the proposed approach can achieve almost 20 BLEU on the same dataset through fine-tuning a pre-trained multi-lingual system in a zero-shot setting.