# Transformers for Low-Resource Languages: Is F'eidir Linn!

arXiv: https://arxiv.org/abs/2403.01985 · HTML: https://mlsanity.com/paper/2403.01985/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 32 references that are in the map.
- First posted on arXiv 4 Mar 2024.

## Builds on (most cited first, 13 of 13)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://mlsanity.com/paper/1808.06226/) (2018)
- [On the Properties of Neural Machine Translation: Encoder-Decoder Approaches](https://mlsanity.com/paper/1409.1259/) (2014)
- [Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates](https://mlsanity.com/paper/1804.10959/) (2018)
- [Quantifying the Carbon Emissions of Machine Learning](https://mlsanity.com/paper/1910.09700/) (2019)
- [OpenNMT: Open-source Toolkit for Neural Machine Translation](https://mlsanity.com/paper/1709.03815/) (2017)
- [Deep Reconstruction-Classification Networks for Unsupervised Domain Adaptation](https://mlsanity.com/paper/1607.03516/) (2016)
- [A Call for Prudent Choice of Subword Merge Operations in Neural Machine Translation](https://mlsanity.com/paper/1905.10453/) (2019)
- [Optimizing Transformer for Low-Resource Neural Machine Translation](https://mlsanity.com/paper/2011.02266/) (2020)
- [DGT-TM: A freely Available Translation Memory in 22 Languages](https://mlsanity.com/paper/1309.5226/) (2013)
- [On Optimal Transformer Depth for Low-Resource Language Translation](https://mlsanity.com/paper/2004.04418/) (2020)

## Built on by (most cited first, 9 of 9)

- [Leveraging Synthetic Audio Data for End-to-End Low-Resource Speech Translation](https://mlsanity.com/paper/2406.17363/) (2024)
- [Is Small Language Model the Silver Bullet to Low-Resource Languages Machine Translation?](https://mlsanity.com/paper/2503.24102/) (2025)
- [gaHealth: An English-Irish Bilingual Corpus of Health Data](https://mlsanity.com/paper/2403.03575/) (2024)
- [Synthetic Fluency: Hallucinations, Confabulations, and the Creation of Irish Words in LLM-Generated Translations](https://mlsanity.com/paper/2504.07680/) (2025)
- [Data driven approaches in nanophotonics: A review of AI-enabled metadevices](https://mlsanity.com/paper/2510.00283/) (2025)
- [SemiAdapt and SemiLoRA: Efficient Domain Adaptation for Transformer-based Low-Resource Language Translation with a Case Study on Irish](https://mlsanity.com/paper/2510.18725/) (2025)
- [Design of an Open-Source Architecture for Neural Machine Translation](https://mlsanity.com/paper/2403.03582/) (2024)
- [Segmentation Beyond Defaults: Asymmetrical Byte Pair Encoding for Optimal Machine Translation Performance](https://mlsanity.com/paper/2511.03383/) (2025)
- [MoirfEolas and Cr'iochScore: Developing Resources for and the Evaluation of Tokenization Alignment with Irish Morphology](https://mlsanity.com/paper/2609.05022/) (2026)

## Abstract (arXiv)

The Transformer model is the state-of-the-art in Machine Translation. However, in general, neural translation models often under perform on language pairs with insufficient training data. As a consequence, relatively few experiments have been carried out using this architecture on low-resource language pairs. In this study, hyperparameter optimization of Transformer models in translating the low-resource English-Irish language pair is evaluated. We demonstrate that choosing appropriate parameters leads to considerable performance improvements. Most importantly, the correct choice of subword model is shown to be the biggest driver of translation performance. SentencePiece models using both unigram and BPE approaches were appraised. Variations on model architectures included modifying the number of layers, testing various regularisation techniques and evaluating the optimal number of heads for attention. A generic 55k DGT corpus and an in-domain 88k public admin corpus were used for evaluation. A Transformer optimized model demonstrated a BLEU score improvement of 7.8 points when compared with a baseline RNN model. Improvements were observed across a range of metrics, including TER, indicating a substantially reduced post editing effort for Transformer optimized models with 16k BPE subword models. Bench-marked against Google Translate, our translation engines demonstrated significant improvements. The question of whether or not Transformers can be used effectively in a low-resource setting of English-Irish translation has been addressed. Is f'eidir linn - yes we can.