# Gl'orIA -- A Generative and Open Large Language Model for Portuguese

arXiv: https://arxiv.org/abs/2402.12969 · HTML: https://mlsanity.com/paper/2402.12969/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 38 references that are in the map.
- First posted on arXiv 20 Feb 2024.

## Builds on (most cited first, 21 of 21)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [mT5: A massively multilingual pre-trained text-to-text transformer](https://mlsanity.com/paper/2010.11934/) (2020)
- [Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling](https://mlsanity.com/paper/2304.01373/) (2023)
- [The LAMBADA dataset: Word prediction requiring a broad discourse context](https://mlsanity.com/paper/1606.06031/) (2016)
- [Efficient Attention: Attention with Linear Complexities](https://mlsanity.com/paper/1812.01243/) (2018)
- [CamemBERT: a Tasty French Language Model](https://mlsanity.com/paper/1911.03894/) (2019)
- [Factuality Enhanced Language Models for Open-Ended Text Generation](https://mlsanity.com/paper/2206.04624/) (2022)
- [Towards a Cleaner Document-Oriented Multilingual Crawled Corpus](https://mlsanity.com/paper/2201.06642/) (2022)
- [mGPT: Few-Shot Learners Go Multilingual](https://mlsanity.com/paper/2204.07580/) (2022)
- [Multilingual is not enough: BERT for Finnish](https://mlsanity.com/paper/1912.07076/) (2019)
- [Increasing Faithfulness in Knowledge-Grounded Dialogue with Controllable Features](https://mlsanity.com/paper/2107.06963/) (2021)
- [PTT5: Pretraining and validating the T5 model on Brazilian Portuguese data](https://mlsanity.com/paper/2008.09144/) (2020)
- [Advancing Neural Encoding of Portuguese with Transformer Albertina PT-*](https://mlsanity.com/paper/2305.06721/) (2023)
- [Plan-Grounded Large Language Models for Dual Goal Conversational Settings](https://mlsanity.com/paper/2402.01053/) (2024)
- [TWIZ-v2: The Wizard of Multimodal Conversational-Stimulus](https://mlsanity.com/paper/2310.02118/) (2023)

## Built on by (most cited first, 16 of 16)

- [Anything Goes? A Crosslinguistic Study of (Im)possible Language Learning in LMs](https://mlsanity.com/paper/2502.18795/) (2025)
- [Tucano: Advancing Neural Text Generation for Portuguese](https://mlsanity.com/paper/2411.07854/) (2024)
- [ptt5-v2: A Closer Look at Continued Pretraining of T5 Models for the Portuguese Language](https://mlsanity.com/paper/2406.10806/) (2024)
- [Building High-Quality Datasets for Portuguese LLMs: From Common Crawl Snapshots to Industrial-Grade Corpora](https://mlsanity.com/paper/2509.08824/) (2025)
- [A Survey of Large Language Models for European Languages](https://mlsanity.com/paper/2408.15040/) (2024)
- [PoETa v2: Toward More Robust Evaluation of Large Language Models in Portuguese](https://mlsanity.com/paper/2511.17808/) (2025)
- [Curi'o-Edu 7B: Examining Data Selection Impacts in LLM Continued Pretraining](https://mlsanity.com/paper/2512.12770/) (2025)
- [Open Generative Large Language Models for Galician](https://mlsanity.com/paper/2406.13893/) (2024)
- [SLIM-RAFT: A Novel Fine-Tuning Approach to Improve Cross-Linguistic Performance for Mercosur Common Nomenclature](https://mlsanity.com/paper/2408.03936/) (2024)
- [ALBA: A European Portuguese Benchmark for Evaluating Language and Linguistic Dimensions in Generative LLMs](https://mlsanity.com/paper/2603.26516/) (2026)
- [LegalBench.PT: A Benchmark for Portuguese Law](https://mlsanity.com/paper/2502.16357/) (2025)
- [A Legal Framework for Natural Language Processing Model Training in Portugal](https://mlsanity.com/paper/2405.00536/) (2024)
- [P3B3: A Multi-Turn Conversational Benchmark for Measuring European and Brazilian Portuguese Variety Bias in LLMs](https://mlsanity.com/paper/2606.16753/) (2026)
- [AMALIA Technical Report: A Fully Open Source Large Language Model for European Portuguese](https://mlsanity.com/paper/2603.26511/) (2026)
- [Language Models for Portuguese: A Systematic Mapping Study](https://mlsanity.com/paper/2608.18138/) (2026)
- [Manac'a-1B: An Open, Reproducible Brazilian-Portuguese Language Model and a Tokenizer-Aware, Paired Evaluation](https://mlsanity.com/paper/2608.30114/) (2026)

## Abstract (arXiv)

Significant strides have been made in natural language tasks, largely attributed to the emergence of powerful large language models (LLMs). These models, pre-trained on extensive and diverse corpora, have become increasingly capable of comprehending the intricacies of language. Despite the abundance of LLMs for many high-resource languages, the availability of such models remains limited for European Portuguese. We introduce Gl'orIA, a robust European Portuguese decoder LLM. To pre-train Gl'orIA, we assembled a comprehensive PT-PT text corpus comprising 35 billion tokens from various sources. We present our pre-training methodology, followed by an assessment of the model's effectiveness on multiple downstream tasks. Additionally, to evaluate our models' language modeling capabilities, we introduce CALAME-PT (Context-Aware LAnguage Modeling Evaluation for Portuguese), the first Portuguese zero-shot language-modeling benchmark. Evaluation shows that Gl'orIA significantly outperforms existing open PT decoder models in language modeling and that it can generate sound, knowledge-rich, and coherent PT-PT text. The model also exhibits strong potential for various downstream tasks.