mlsanityData to 7 Oct 2026

Overcoming Data Scarcity in Generative Language Modelling for Low-Resource Languages: A Systematic Review

arXiv 2505.04531 ↗First posted 7 May 2025cs.CL cs.AIcited by 8

Not on the map. Why?

Abstract (arXiv)

Generative language modelling has surged in popularity with the emergence of services such as ChatGPT and Google Gemini. While these models have demonstrated transformative potential in productivity and communication, they overwhelmingly cater to high-resource languages like English. This has amplified concerns over linguistic inequality in natural language processing (NLP). This paper presents the first systematic review focused specifically on strategies to address data scarcity in generative language modelling for low-resource languages (LRL). Drawing from 54 studies, we identify, categorise and evaluate technical approaches, including monolingual data augmentation, back-translation, multilingual training, and prompt engineering, across generative tasks. We also analyse trends in architecture choices, language family representation, and evaluation methods. Our findings highlight a strong reliance on transformer-based models, a concentration on a small subset of LRLs, and a lack of consistent evaluation across studies. We conclude with recommendations for extending these methods to a wider range of LRLs and outline open challenges in building equitable generative language systems. Ultimately, this review aims to support researchers and developers in building inclusive AI tools for underrepresented languages, a necessary step toward empowering LRL speakers and the preservation of linguistic diversity in a world increasingly shaped by large-scale language technologies.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Builds on 37 of its 123 references that are in the map. First posted on arXiv 7 May 2025.

Builds on

37 of 123 references.

References · 37

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2024The Llama 3 Herd of Modelscited by 14,192
  4. 2020Scaling Laws for Neural Language Modelscited by 6,411
  5. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  6. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  7. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
  8. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
  9. 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,028
  10. 2020The State and Fate of Linguistic Diversity and Inclusion in the NLP Worldcited by 789
  11. 2023ChatGPT Beyond English: Towards a Comprehensive Evaluation of Large Language Models in Multilingual Learningcited by 203
  12. 2024Aya Dataset: An Open-Access Collection for Multilingual Instruction Tuningcited by 174
25 more
  1. 2017Data Augmentation for Low-Resource Neural Machine Translationcited by 139
  2. 2020Low-resource Languages: A Review of Past Work and Future Challengescited by 90
  3. 2023Personalisation within bounds: A risk taxonomy and policy framework for the alignment of large language models with personalised feedbackcited by 86
  4. 2018SwitchOut: an Efficient Data Augmentation Algorithm for Neural Machine Translationcited by 72
  5. 2019Soft Contextual Data Augmentation for Neural Machine Translationcited by 49
  6. 2020Paraphrase Augmented Task-Oriented Dialog Generationcited by 43
  7. 2022A Survey on Cross-Lingual Summarizationcited by 38
  8. 2020Open Source Software for Efficient and Transparent Reviewscited by 34
  9. 2024Teaching Large Language Models an Unseen Language on the Flycited by 29
  10. 2020ChrEn: Cherokee-English Machine Translation for Endangered Language Revitalizationcited by 20
  11. 2022Cedille: A large autoregressive French language modelcited by 20
  12. 2020GePpeTto Carves Italian into a Language Modelcited by 16
  13. 2021Meta Back-translationcited by 16
  14. 2021A Survey on Low-Resource Neural Machine Translationcited by 16
  15. 2019Dual Supervised Learning for Natural Language Understanding and Generationcited by 14
  16. 2023QASiNa: Religious Domain Question Answering using Sirah Nabawiyahcited by 14
  17. 2024Bridging the Bosphorus: Advancing Turkish Large Language Models through Strategies for Low-Resource Language Adaptation and Benchmarkingcited by 10
  18. 2022Sequence to sequence pretraining for a less-resourced Slovenian languagecited by 9
  19. 2024Targeted Multilingual Adaptation for Low-resource Language Familiescited by 9
  20. 2024LlamaTurk: Adapting Open-Source Generative Large Language Models for Low-Resource Languagecited by 6
  21. 2024BEnQA: A Question Answering and Reasoning Benchmark for Bengali and Englishcited by 4
  22. 2024Bridging the gap in online hate speech detection: a comparative analysis of BERT and traditional models for homophobic content identification on X/Twittercited by 4
  23. 2024Improving Cross-Domain Low-Resource Text Generation through LLM Post-Editing: A Programmer-Interpreter Approachcited by 3
  24. 2023Revitalizing Endangered Languages: AI-powered language learning as a catalyst for language appreciationcited by 2
  25. 2023Generative Pre-trained Transformer for Vietnamese Community-based COVID-19 Question Answeringcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 4; 2026: 4.

8 papers.

  1. 2025Learning-at-Criticality in Large Language Models for Quantum Field Theory and Beyondcited by 5
  2. 2025Irish-BLiMP: A Linguistic Benchmark for Evaluating Human and Language Model Performance in a Low-Resource Settingcited by 2
  3. 2025SemiAdapt and SemiLoRA: Efficient Domain Adaptation for Transformer-based Low-Resource Language Translation with a Case Study on Irishcited by 1
  4. 2025FastPOS: Language-Agnostic Scalable POS Tagging Framework Low-Resource Use Casecited by 0
  5. 2026Targeted Syntactic Evaluation of Language Models on Georgian Case Alignmentcited by 0
  6. 2026Convex Dataset Valuation for Post-Trainingcited by 0
  7. 2026Bridging the Stability-Expressivity Gap: Synthetic Data Scaling and Preference Alignment for Low-Resource Spoken Language Modelscited by 0
  8. 2026SemiAdapt-Instruct: Extensible Instruction Tuning via Latent Domain-Specialised Adapterscited by 0