# Open Generative Large Language Models for Galician

arXiv: https://arxiv.org/abs/2406.13893 · HTML: https://mlsanity.com/paper/2406.13893/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 24 references that are in the map.
- First posted on arXiv 19 Jun 2024.

## Builds on (most cited first, 17 of 17)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering](https://mlsanity.com/paper/1809.02789/) (2018)
- [ZeRO: Memory Optimizations Toward Training Trillion Parameter Models](https://mlsanity.com/paper/1910.02054/) (2019)
- [The LAMBADA dataset: Word prediction requiring a broad discourse context](https://mlsanity.com/paper/1606.06031/) (2016)
- [Neural Network Acceptability Judgments](https://mlsanity.com/paper/1805.12471/) (2018)
- [Measuring Coding Challenge Competence With APPS](https://mlsanity.com/paper/2105.09938/) (2021)
- [The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variants](https://mlsanity.com/paper/2308.16884/) (2023)
- [Continual Pre-training of Language Models](https://mlsanity.com/paper/2302.03241/) (2023)
- [Continual Pre-Training of Large Language Models: How to (re)warm your model?](https://mlsanity.com/paper/2308.04014/) (2023)
- [Latxa: An Open Language Model and Evaluation Suite for Basque](https://mlsanity.com/paper/2403.20266/) (2024)
- [Evaluating the Diversity, Equity and Inclusion of NLP Technology: A Case Study for Indian Languages](https://mlsanity.com/paper/2205.12676/) (2022)
- [Advancing Generative AI for Portuguese with Open Decoder Gerv'asio PT*](https://mlsanity.com/paper/2402.18766/) (2024)
- [Embedding structure matters: Comparing methods to adapt multilingual vocabularies to new languages](https://mlsanity.com/paper/2309.04679/) (2023)
- [Gl'orIA -- A Generative and Open Large Language Model for Portuguese](https://mlsanity.com/paper/2402.12969/) (2024)
- [Exploring the Representation of Word Meanings in Context: A Case Study on Homonymy and Synonymy](https://mlsanity.com/paper/2106.13553/) (2021)
- [Bertinho: Galician BERT Representations](https://mlsanity.com/paper/2103.13799/) (2021)

## Built on by (most cited first, 3 of 3)

- [RoQLlama: A Lightweight Romanian Adapted Language Model](https://mlsanity.com/paper/2410.04269/) (2024)
- [Whisper-LM: Improving ASR Models with Language Models for Low-Resource Languages](https://mlsanity.com/paper/2503.23542/) (2025)
- [La Leaderboard: A Large Language Model Leaderboard for Spanish Varieties and Languages of Spain and Latin America](https://mlsanity.com/paper/2507.00999/) (2025)

## Abstract (arXiv)

Large language models (LLMs) have transformed natural language processing. Yet, their predominantly English-centric training has led to biases and performance disparities across languages. This imbalance marginalizes minoritized languages, making equitable access to NLP technologies more difficult for languages with lower resources, such as Galician. We present the first two generative LLMs focused on Galician to bridge this gap. These models, freely available as open-source resources, were trained using a GPT architecture with 1.3B parameters on a corpus of 2.1B words. Leveraging continual pretraining, we adapt to Galician two existing LLMs trained on larger corpora, thus mitigating the data constraints that would arise if the training were performed from scratch. The models were evaluated using human judgments and task-based datasets from standardized benchmarks. These evaluations reveal a promising performance, underscoring the importance of linguistic diversity in generative models.