mlsanityData to 7 Oct 2026

MarIA: Spanish Language Models

arXiv 2107.07253 ↗First posted 15 Jul 2021cs.CL cs.AIcited by 36

Not on the map. Why?

Abstract (arXiv)

This work presents MarIA, a family of Spanish language models and associated resources made available to the industry and the research community. Currently, MarIA includes RoBERTa-base, RoBERTa-large, GPT2 and GPT2-large Spanish language models, which can arguably be presented as the largest and most proficient language models in Spanish. The models were pretrained using a massive corpus of 570GB of clean and deduplicated texts with 135 billion words extracted from the Spanish Web Archive crawled by the National Library of Spain between 2009 and 2019. We assessed the performance of the models with nine existing evaluation datasets and with a novel extractive Question Answering dataset created ex novo. Overall, MarIA models outperform the existing Spanish models across a variety of NLU tasks and training settings.

Key facts. Cited by 36 papers in the map (data through 7 Oct 2026). Builds on 24 of its 52 references that are in the map. First posted on arXiv 15 Jul 2021.

Builds on

24 of 52 references.

References · 24

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,346
  6. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  7. 2018Deep contextualized word representationscited by 2,285
  8. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
  9. 2016Enriching Word Vectors with Subword Informationcited by 1,578
  10. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  11. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 913
  12. 2019On the Cross-lingual Transferability of Monolingual Representationscited by 575
12 more
  1. 2020Scaling Laws for Autoregressive Generative Modelingcited by 510
  2. 2019CamemBERT: a Tasty French Language Modelcited by 368
  3. 2019PAWS-X: A Cross-lingual Adversarial Dataset for Paraphrase Identificationcited by 262
  4. 2019FlauBERT: Unsupervised Language Model Pre-training for Frenchcited by 153
  5. 2020PhoBERT: Pre-trained language models for Vietnamesecited by 148
  6. 2019Multilingual is not enough: BERT for Finnishcited by 118
  7. 2019BERTje: A Dutch BERT Modelcited by 109
  8. 2020What the [MASK]? Making Sense of Language-Specific BERT Modelscited by 44
  9. 2018A Corpus for Multilingual Document Classification in Eight Languagescited by 42
  10. 2019One Epoch Is All You Needcited by 41
  11. 2020Overview of CAPITEL Shared Tasks at IberLEF 2020: Named Entity Recognition and Universal Dependencies Parsingcited by 4
  12. 2021Spanish Biomedical and Clinical Language Embeddingscited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 7; 2023: 11; 2024: 12; 2025: 5; 2026: 1.

36 papers.

Sort:

  1. 2024Aya Model: An Instruction Finetuned Open-Access Multilingual Language Modelcited by 292
  2. 2023Generative AI Text Classification using Ensemble LLM Approachescited by 29
  3. 2024Goldfish: Monolingual Language Models for 350 Languagescited by 27
  4. 2023Advancing Neural Encoding of Portuguese with Transformer Albertina PT-*cited by 24
  5. 2023Selective Pre-training for Private Fine-tuningcited by 22
  6. 2024Multi-Task Contrastive Learning for 8192-Token Bilingual Text Embeddingscited by 20
  7. 2023Leveraging Large Language Models for Topic Classification in the Domain of Public Affairscited by 13
  8. 2024TeenyTinyLlama: open-source tiny language models trained in Brazilian Portuguesecited by 12
  9. 2022Assessing the impact of contextual information in hate speech detectioncited by 10
  10. 2025Llama-3-Nanda-10B-Chat: An Open Generative Large Language Model for Hindicited by 10
  11. 2024Exploring Large Language Models to generate Easy to Read contentcited by 9
  12. 2022Lessons learned from the evaluation of Spanish Language Modelscited by 8
24 more
  1. 2022ALBETO and DistilBETO: Lightweight Spanish Language Modelscited by 6
  2. 2023Explanatory Argument Extraction of Correct Answers in Resident Medical Examscited by 6
  3. 2024A Survey of Large Language Models for European Languagescited by 6
  4. 2025Benchmarking Graph Neural Networks for Document Layout Analysis in Public Affairscited by 6
  5. 2022Leveraging a New Spanish Corpus for Multilingual and Crosslingual Metaphor Detectioncited by 5
  6. 2023Overview of GUA-SPA at IberLEF 2023: Guarani-Spanish Code Switching Analysiscited by 5
  7. 2022End-to-end Multilingual Coreference Resolution with Mention Head Predictioncited by 4
  8. 2023Deep Learning Approaches to Lexical Simplification: A Surveycited by 4
  9. 2024Exploring Multiple Strategies to Improve Multilingual Coreference Resolution in CorefUDcited by 3
  10. 2024An evaluation of LLM code generation capabilities through graded exercisescited by 3
  11. 2023Sequence-to-Sequence Spanish Pre-trained Language Modelscited by 2
  12. 2025Optimal word order for non-causal text generation with Large Language Models: the Spanish casecited by 2
  13. 2022esCorpius: A Massive Spanish Crawling Corpuscited by 1
  14. 2022A Spanish dataset for Targeted Sentiment Analysis of political headlinescited by 1
  15. 2025Predicting potentially abusive clauses in Chilean terms of services with natural language processingcited by 1
  16. 2023A Framework for Identifying Depression on Social Media: MentalRiskES@IberLEF 2023cited by 0
  17. 2023A Cost-aware Study of Depression Language on Social Media using Topic and Affect Contextualizationcited by 0
  18. 2023Language Model Knowledge Distillation for Efficient Question Answering in Spanishcited by 0
  19. 2024Adapting PromptORE for Modern History: Information Extraction from Hispanic Monarchy Documents of the XVIth Centurycited by 0
  20. 2024Identification of emotions on Twitter during the 2022 electoral process in Colombiacited by 0
  21. 2024Predictability and Causality in Spanish and English Natural Language Generationcited by 0
  22. 2024Word Sense Disambiguation in Native Spanish: A Comprehensive Lexical Evaluation Resourcecited by 0
  23. 2025Fine-tuning of Large Language Models for Constituency Parsing Using a Sequence to Sequence Approachcited by 0
  24. 2026Improving Mental Health Screening and Early Risk Detection in Spanishcited by 0