# MarIA: Spanish Language Models

arXiv: https://arxiv.org/abs/2107.07253 · HTML: https://mlsanity.com/paper/2107.07253/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 52 references that are in the map.
- First posted on arXiv 15 Jul 2021.

## Builds on (most cited first, 24 of 24)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [mT5: A massively multilingual pre-trained text-to-text transformer](https://mlsanity.com/paper/2010.11934/) (2020)
- [Enriching Word Vectors with Subword Information](https://mlsanity.com/paper/1607.04606/) (2016)
- [fairseq: A Fast, Extensible Toolkit for Sequence Modeling](https://mlsanity.com/paper/1904.01038/) (2019)
- [XNLI: Evaluating Cross-lingual Sentence Representations](https://mlsanity.com/paper/1809.05053/) (2018)
- [On the Cross-lingual Transferability of Monolingual Representations](https://mlsanity.com/paper/1910.11856/) (2019)
- [Scaling Laws for Autoregressive Generative Modeling](https://mlsanity.com/paper/2010.14701/) (2020)
- [CamemBERT: a Tasty French Language Model](https://mlsanity.com/paper/1911.03894/) (2019)
- [PAWS-X: A Cross-lingual Adversarial Dataset for Paraphrase Identification](https://mlsanity.com/paper/1908.11828/) (2019)
- [FlauBERT: Unsupervised Language Model Pre-training for French](https://mlsanity.com/paper/1912.05372/) (2019)
- [PhoBERT: Pre-trained language models for Vietnamese](https://mlsanity.com/paper/2003.00744/) (2020)
- [Multilingual is not enough: BERT for Finnish](https://mlsanity.com/paper/1912.07076/) (2019)
- [BERTje: A Dutch BERT Model](https://mlsanity.com/paper/1912.09582/) (2019)
- [What the [MASK]? Making Sense of Language-Specific BERT Models](https://mlsanity.com/paper/2003.02912/) (2020)
- [A Corpus for Multilingual Document Classification in Eight Languages](https://mlsanity.com/paper/1805.09821/) (2018)
- [One Epoch Is All You Need](https://mlsanity.com/paper/1906.06669/) (2019)
- [Overview of CAPITEL Shared Tasks at IberLEF 2020: Named Entity Recognition and Universal Dependencies Parsing](https://mlsanity.com/paper/2011.05932/) (2020)
- [Spanish Biomedical and Clinical Language Embeddings](https://mlsanity.com/paper/2102.12843/) (2021)

## Built on by (most cited first, 30 of 36)

- [Aya Model: An Instruction Finetuned Open-Access Multilingual Language Model](https://mlsanity.com/paper/2402.07827/) (2024)
- [Generative AI Text Classification using Ensemble LLM Approaches](https://mlsanity.com/paper/2309.07755/) (2023)
- [Goldfish: Monolingual Language Models for 350 Languages](https://mlsanity.com/paper/2408.10441/) (2024)
- [Advancing Neural Encoding of Portuguese with Transformer Albertina PT-*](https://mlsanity.com/paper/2305.06721/) (2023)
- [Selective Pre-training for Private Fine-tuning](https://mlsanity.com/paper/2305.13865/) (2023)
- [Multi-Task Contrastive Learning for 8192-Token Bilingual Text Embeddings](https://mlsanity.com/paper/2402.17016/) (2024)
- [Leveraging Large Language Models for Topic Classification in the Domain of Public Affairs](https://mlsanity.com/paper/2306.02864/) (2023)
- [TeenyTinyLlama: open-source tiny language models trained in Brazilian Portuguese](https://mlsanity.com/paper/2401.16640/) (2024)
- [Assessing the impact of contextual information in hate speech detection](https://mlsanity.com/paper/2210.00465/) (2022)
- [Llama-3-Nanda-10B-Chat: An Open Generative Large Language Model for Hindi](https://mlsanity.com/paper/2504.06011/) (2025)
- [Exploring Large Language Models to generate Easy to Read content](https://mlsanity.com/paper/2407.20046/) (2024)
- [Lessons learned from the evaluation of Spanish Language Models](https://mlsanity.com/paper/2212.08390/) (2022)
- [Benchmarking Graph Neural Networks for Document Layout Analysis in Public Affairs](https://mlsanity.com/paper/2505.14699/) (2025)
- [ALBETO and DistilBETO: Lightweight Spanish Language Models](https://mlsanity.com/paper/2204.09145/) (2022)
- [Explanatory Argument Extraction of Correct Answers in Resident Medical Exams](https://mlsanity.com/paper/2312.00567/) (2023)
- [A Survey of Large Language Models for European Languages](https://mlsanity.com/paper/2408.15040/) (2024)
- [Leveraging a New Spanish Corpus for Multilingual and Crosslingual Metaphor Detection](https://mlsanity.com/paper/2210.10358/) (2022)
- [Overview of GUA-SPA at IberLEF 2023: Guarani-Spanish Code Switching Analysis](https://mlsanity.com/paper/2309.06163/) (2023)
- [End-to-end Multilingual Coreference Resolution with Mention Head Prediction](https://mlsanity.com/paper/2209.12516/) (2022)
- [Deep Learning Approaches to Lexical Simplification: A Survey](https://mlsanity.com/paper/2305.12000/) (2023)
- [Exploring Multiple Strategies to Improve Multilingual Coreference Resolution in CorefUD](https://mlsanity.com/paper/2408.16893/) (2024)
- [An evaluation of LLM code generation capabilities through graded exercises](https://mlsanity.com/paper/2410.16292/) (2024)
- [Sequence-to-Sequence Spanish Pre-trained Language Models](https://mlsanity.com/paper/2309.11259/) (2023)
- [Optimal word order for non-causal text generation with Large Language Models: the Spanish case](https://mlsanity.com/paper/2502.14451/) (2025)
- [esCorpius: A Massive Spanish Crawling Corpus](https://mlsanity.com/paper/2206.15147/) (2022)
- [A Spanish dataset for Targeted Sentiment Analysis of political headlines](https://mlsanity.com/paper/2208.13947/) (2022)
- [Predicting potentially abusive clauses in Chilean terms of services with natural language processing](https://mlsanity.com/paper/2502.00865/) (2025)
- [A Framework for Identifying Depression on Social Media: MentalRiskES@IberLEF 2023](https://mlsanity.com/paper/2306.16125/) (2023)
- [A Cost-aware Study of Depression Language on Social Media using Topic and Affect Contextualization](https://mlsanity.com/paper/2306.17564/) (2023)
- [Language Model Knowledge Distillation for Efficient Question Answering in Spanish](https://mlsanity.com/paper/2312.04193/) (2023)

## Abstract (arXiv)

This work presents MarIA, a family of Spanish language models and associated resources made available to the industry and the research community. Currently, MarIA includes RoBERTa-base, RoBERTa-large, GPT2 and GPT2-large Spanish language models, which can arguably be presented as the largest and most proficient language models in Spanish. The models were pretrained using a massive corpus of 570GB of clean and deduplicated texts with 135 billion words extracted from the Spanish Web Archive crawled by the National Library of Spain between 2009 and 2019. We assessed the performance of the models with nine existing evaluation datasets and with a novel extractive Question Answering dataset created ex novo. Overall, MarIA models outperform the existing Spanish models across a variety of NLU tasks and training settings.