# Text Generation Models for Luxembourgish with Limited Data: A Balanced Multilingual Strategy

arXiv: https://arxiv.org/abs/2412.09415 · HTML: https://mlsanity.com/paper/2412.09415/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 59 references that are in the map.
- First posted on arXiv 12 Dec 2024.

## Builds on (most cited first, 22 of 22)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [mT5: A massively multilingual pre-trained text-to-text transformer](https://mlsanity.com/paper/2010.11934/) (2020)
- [BLEURT: Learning Robust Metrics for Text Generation](https://mlsanity.com/paper/2004.04696/) (2020)
- [ByT5: Towards a token-free future with pre-trained byte-to-byte models](https://mlsanity.com/paper/2105.13626/) (2021)
- [Benchmarking Large Language Models for News Summarization](https://mlsanity.com/paper/2301.13848/) (2023)
- [Sentiment Analysis in the Era of Large Language Models: A Reality Check](https://mlsanity.com/paper/2305.15005/) (2023)
- [FlauBERT: Unsupervised Language Model Pre-training for French](https://mlsanity.com/paper/1912.05372/) (2019)
- [German's Next Language Model](https://mlsanity.com/paper/2010.10906/) (2020)
- [Systematic Inequalities in Language Technology Performance across the World's Languages](https://mlsanity.com/paper/2110.06733/) (2021)
- [ConvBERT: Improving BERT with Span-based Dynamic Convolution](https://mlsanity.com/paper/2008.02496/) (2020)
- [On Learning to Summarize with Large Language Models as References](https://mlsanity.com/paper/2305.14239/) (2023)
- [A Comprehensive Survey on Applications of Transformers for Deep Learning Tasks](https://mlsanity.com/paper/2306.07303/) (2023)
- [AraT5: Text-to-Text Transformers for Arabic Language Generation](https://mlsanity.com/paper/2109.12068/) (2021)
- [PTT5: Pretraining and validating the T5 model on Brazilian Portuguese data](https://mlsanity.com/paper/2008.09144/) (2020)
- [A Family of Pretrained Transformer Language Models for Russian](https://mlsanity.com/paper/2309.10931/) (2023)
- [Model Selection for Cross-Lingual Transfer](https://mlsanity.com/paper/2010.06127/) (2020)
- [V=arta: A Large-Scale Headline-Generation Dataset for Indic Languages](https://mlsanity.com/paper/2305.05858/) (2023)
- [NSINA: A News Corpus for Sinhala](https://mlsanity.com/paper/2403.16571/) (2024)

## Built on by (most cited first, 7 of 7)

- [Is Small Language Model the Silver Bullet to Low-Resource Languages Machine Translation?](https://mlsanity.com/paper/2503.24102/) (2025)
- [Testing Low-Resource Language Support in LLMs Using Language Proficiency Exams: the Case of Luxembourgish](https://mlsanity.com/paper/2504.01667/) (2025)
- [LuxInstruct: A Cross-Lingual Instruction Tuning Dataset For Luxembourgish](https://mlsanity.com/paper/2510.07074/) (2025)
- [LuxIT: A Luxembourgish Instruction Tuning Dataset from Monolingual Seed Data](https://mlsanity.com/paper/2510.24434/) (2025)
- [A Subword Embedding Approach for Variation Detection in Luxembourgish User Comments](https://mlsanity.com/paper/2602.11795/) (2026)
- [Variation is the Norm: Embracing Sociolinguistics in NLP](https://mlsanity.com/paper/2603.24222/) (2026)
- [Language Ideologies in a Multilingual Society: An LLM-based Analysis of Luxembourgish News Comments](https://mlsanity.com/paper/2604.27661/) (2026)

## Abstract (arXiv)

This paper addresses the challenges in developing language models for less-represented languages, with a focus on Luxembourgish. Despite its active development, Luxembourgish faces a digital data scarcity, exacerbated by Luxembourg's multilingual context. We propose a novel text generation model based on the T5 architecture, combining limited Luxembourgish data with equal amounts, in terms of size and type, of German and French data. We hypothesise that a model trained on Luxembourgish, German, and French will improve the model's cross-lingual transfer learning capabilities and outperform monolingual and large multilingual models. To verify this, the study at hand explores whether multilingual or monolingual training is more beneficial for Luxembourgish language generation. For the evaluation, we introduce LuxGen, a text generation benchmark that is the first of its kind for Luxembourgish.