# Optimizing Transformer for Low-Resource Neural Machine Translation

arXiv: https://arxiv.org/abs/2011.02266 · HTML: https://mlsanity.com/paper/2011.02266/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Nov 2020.

## Built on by (most cited first, 24 of 24)

- [Cramming: Training a Language Model on a Single GPU in One Day](https://mlsanity.com/paper/2212.14034/) (2022)
- [Long-Range Transformers for Dynamic Spatiotemporal Forecasting](https://mlsanity.com/paper/2109.12218/) (2021)
- [Sign Language Translation from Instructional Videos](https://mlsanity.com/paper/2304.06371/) (2023)
- [Transformer-based Models to Deal with Heterogeneous Environments in Human Activity Recognition](https://mlsanity.com/paper/2209.11750/) (2022)
- [Machine Translation in the Covid domain: an English-Irish case study for LoResMT 2021](https://mlsanity.com/paper/2403.01196/) (2024)
- [Transformers for Low-Resource Languages: Is F'eidir Linn!](https://mlsanity.com/paper/2403.01985/) (2024)
- [Subword Segmental Machine Translation: Unifying Segmentation and Target Sentence Generation](https://mlsanity.com/paper/2305.07005/) (2023)
- [On Biasing Transformer Attention Towards Monotonicity](https://mlsanity.com/paper/2104.03945/) (2021)
- [Low-Resourced Machine Translation for Senegalese Wolof Language](https://mlsanity.com/paper/2305.00606/) (2023)
- [Beqi: Revitalize the Senegalese Wolof Language with a Robust Spelling Corrector](https://mlsanity.com/paper/2305.08518/) (2023)
- [Core Building Blocks: Next Gen Geo Spatial GPT Application](https://mlsanity.com/paper/2310.11029/) (2023)
- [Zero-pronoun Data Augmentation for Japanese-to-English Translation](https://mlsanity.com/paper/2107.00318/) (2021)
- [How Effective is Byte Pair Encoding for Out-Of-Vocabulary Words in Neural Machine Translation?](https://mlsanity.com/paper/2208.05225/) (2022)
- [Human Evaluation of English--Irish Transformer-Based NMT](https://mlsanity.com/paper/2403.02366/) (2024)
- [Small Batch Sizes Improve Training of Low-Resource Neural MT](https://mlsanity.com/paper/2203.10579/) (2022)
- [adaptNMT: an open-source, language-agnostic development environment for Neural Machine Translation](https://mlsanity.com/paper/2403.02367/) (2024)
- [Linguistically-driven Multi-task Pre-training for Low-resource Neural Machine Translation](https://mlsanity.com/paper/2201.08070/) (2022)
- [PICT@DravidianLangTech-ACL2022: Neural Machine Translation On Dravidian Languages](https://mlsanity.com/paper/2204.09098/) (2022)
- [High-Resource Methodological Bias in Low-Resource Investigations](https://mlsanity.com/paper/2211.07534/) (2022)
- [Joint Dropout: Improving Generalizability in Low-Resource Neural Machine Translation through Phrase Pair Variables](https://mlsanity.com/paper/2307.12835/) (2023)
- [Effective General-Domain Data Inclusion for the Machine Translation Task by Vanilla Transformers](https://mlsanity.com/paper/2209.14073/) (2022)
- [Impact of Domain-Adapted Multilingual Neural Machine Translation in the Medical Domain](https://mlsanity.com/paper/2212.02143/) (2022)
- [Attention Link: An Efficient Attention-Based Low Resource Machine Translation Architecture](https://mlsanity.com/paper/2302.00340/) (2023)
- [Statistical Machine Translation Systems of English-Pnar Language Pair : Some Insights of the Emperical Study](https://mlsanity.com/paper/2608.23120/) (2026)

## Abstract (arXiv)

Language pairs with limited amounts of parallel data, also known as low-resource languages, remain a challenge for neural machine translation. While the Transformer model has achieved significant improvements for many language pairs and has become the de facto mainstream architecture, its capability under low-resource conditions has not been fully investigated yet. Our experiments on different subsets of the IWSLT14 training data show that the effectiveness of Transformer under low-resource conditions is highly dependent on the hyper-parameter settings. Our experiments show that using an optimized Transformer for low-resource conditions improves the translation quality up to 7.3 BLEU points compared to using the Transformer default settings.