# Overcoming Data Scarcity in Generative Language Modelling for Low-Resource Languages: A Systematic Review

arXiv: https://arxiv.org/abs/2505.04531 · HTML: https://mlsanity.com/paper/2505.04531/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 37 of its 123 references that are in the map.
- First posted on arXiv 7 May 2025.

## Builds on (most cited first, 30 of 37)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Training Compute-Optimal Large Language Models](https://mlsanity.com/paper/2203.15556/) (2022)
- [BLOOM: A 176B-Parameter Open-Access Multilingual Language Model](https://mlsanity.com/paper/2211.05100/) (2022)
- [mT5: A massively multilingual pre-trained text-to-text transformer](https://mlsanity.com/paper/2010.11934/) (2020)
- [A Call for Clarity in Reporting BLEU Scores](https://mlsanity.com/paper/1804.08771/) (2018)
- [Multilingual Denoising Pre-training for Neural Machine Translation](https://mlsanity.com/paper/2001.08210/) (2020)
- [The State and Fate of Linguistic Diversity and Inclusion in the NLP World](https://mlsanity.com/paper/2004.09095/) (2020)
- [ChatGPT Beyond English: Towards a Comprehensive Evaluation of Large Language Models in Multilingual Learning](https://mlsanity.com/paper/2304.05613/) (2023)
- [Aya Dataset: An Open-Access Collection for Multilingual Instruction Tuning](https://mlsanity.com/paper/2402.06619/) (2024)
- [Data Augmentation for Low-Resource Neural Machine Translation](https://mlsanity.com/paper/1705.00440/) (2017)
- [Low-resource Languages: A Review of Past Work and Future Challenges](https://mlsanity.com/paper/2006.07264/) (2020)
- [Personalisation within bounds: A risk taxonomy and policy framework for the alignment of large language models with personalised feedback](https://mlsanity.com/paper/2303.05453/) (2023)
- [SwitchOut: an Efficient Data Augmentation Algorithm for Neural Machine Translation](https://mlsanity.com/paper/1808.07512/) (2018)
- [Soft Contextual Data Augmentation for Neural Machine Translation](https://mlsanity.com/paper/1905.10523/) (2019)
- [Paraphrase Augmented Task-Oriented Dialog Generation](https://mlsanity.com/paper/2004.07462/) (2020)
- [A Survey on Cross-Lingual Summarization](https://mlsanity.com/paper/2203.12515/) (2022)
- [Open Source Software for Efficient and Transparent Reviews](https://mlsanity.com/paper/2006.12166/) (2020)
- [Teaching Large Language Models an Unseen Language on the Fly](https://mlsanity.com/paper/2402.19167/) (2024)
- [ChrEn: Cherokee-English Machine Translation for Endangered Language Revitalization](https://mlsanity.com/paper/2010.04791/) (2020)
- [Cedille: A large autoregressive French language model](https://mlsanity.com/paper/2202.03371/) (2022)
- [GePpeTto Carves Italian into a Language Model](https://mlsanity.com/paper/2004.14253/) (2020)
- [Meta Back-translation](https://mlsanity.com/paper/2102.07847/) (2021)
- [A Survey on Low-Resource Neural Machine Translation](https://mlsanity.com/paper/2107.04239/) (2021)
- [QASiNa: Religious Domain Question Answering using Sirah Nabawiyah](https://mlsanity.com/paper/2310.08102/) (2023)
- [Dual Supervised Learning for Natural Language Understanding and Generation](https://mlsanity.com/paper/1905.06196/) (2019)
- [Bridging the Bosphorus: Advancing Turkish Large Language Models through Strategies for Low-Resource Language Adaptation and Benchmarking](https://mlsanity.com/paper/2405.04685/) (2024)
- [Sequence to sequence pretraining for a less-resourced Slovenian language](https://mlsanity.com/paper/2207.13988/) (2022)

## Built on by (most cited first, 8 of 8)

- [Learning-at-Criticality in Large Language Models for Quantum Field Theory and Beyond](https://mlsanity.com/paper/2506.03703/) (2025)
- [Irish-BLiMP: A Linguistic Benchmark for Evaluating Human and Language Model Performance in a Low-Resource Setting](https://mlsanity.com/paper/2510.20957/) (2025)
- [SemiAdapt and SemiLoRA: Efficient Domain Adaptation for Transformer-based Low-Resource Language Translation with a Case Study on Irish](https://mlsanity.com/paper/2510.18725/) (2025)
- [FastPOS: Language-Agnostic Scalable POS Tagging Framework Low-Resource Use Case](https://mlsanity.com/paper/2512.00745/) (2025)
- [Targeted Syntactic Evaluation of Language Models on Georgian Case Alignment](https://mlsanity.com/paper/2602.10661/) (2026)
- [Convex Dataset Valuation for Post-Training](https://mlsanity.com/paper/2605.16704/) (2026)
- [Bridging the Stability-Expressivity Gap: Synthetic Data Scaling and Preference Alignment for Low-Resource Spoken Language Models](https://mlsanity.com/paper/2605.27383/) (2026)
- [SemiAdapt-Instruct: Extensible Instruction Tuning via Latent Domain-Specialised Adapters](https://mlsanity.com/paper/2608.05161/) (2026)

## Abstract (arXiv)

Generative language modelling has surged in popularity with the emergence of services such as ChatGPT and Google Gemini. While these models have demonstrated transformative potential in productivity and communication, they overwhelmingly cater to high-resource languages like English. This has amplified concerns over linguistic inequality in natural language processing (NLP). This paper presents the first systematic review focused specifically on strategies to address data scarcity in generative language modelling for low-resource languages (LRL). Drawing from 54 studies, we identify, categorise and evaluate technical approaches, including monolingual data augmentation, back-translation, multilingual training, and prompt engineering, across generative tasks. We also analyse trends in architecture choices, language family representation, and evaluation methods. Our findings highlight a strong reliance on transformer-based models, a concentration on a small subset of LRLs, and a lack of consistent evaluation across studies. We conclude with recommendations for extending these methods to a wider range of LRLs and outline open challenges in building equitable generative language systems. Ultimately, this review aims to support researchers and developers in building inclusive AI tools for underrepresented languages, a necessary step toward empowering LRL speakers and the preservation of linguistic diversity in a world increasingly shaped by large-scale language technologies.