# Lessons learned from the evaluation of Spanish Language Models arXiv: https://arxiv.org/abs/2212.08390 ยท HTML: https://mlsanity.com/paper/2212.08390/ ## Key facts - Cited by 8 papers in the map (data through 14 Oct 2026). - Builds on 34 of its 54 references that are in the map. - First posted on arXiv 16 Dec 2022. ## Builds on (most cited first, 30 of 34) - [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018) - [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020) - [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019) - [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019) - [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019) - [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022) - [BLOOM: A 176B-Parameter Open-Access Multilingual Language Model](https://mlsanity.com/paper/2211.05100/) (2022) - [mT5: A massively multilingual pre-trained text-to-text transformer](https://mlsanity.com/paper/2010.11934/) (2020) - [DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing](https://mlsanity.com/paper/2111.09543/) (2021) - [XNLI: Evaluating Cross-lingual Sentence Representations](https://mlsanity.com/paper/1809.05053/) (2018) - [How multilingual is Multilingual BERT?](https://mlsanity.com/paper/1906.01502/) (2019) - [CamemBERT: a Tasty French Language Model](https://mlsanity.com/paper/1911.03894/) (2019) - [PAWS-X: A Cross-lingual Adversarial Dataset for Paraphrase Identification](https://mlsanity.com/paper/1908.11828/) (2019) - [Quality at a Glance: An Audit of Web-Crawled Multilingual Datasets](https://mlsanity.com/paper/2103.12028/) (2021) - [Are All Languages Created Equal in Multilingual BERT?](https://mlsanity.com/paper/2005.09093/) (2020) - [Better Fine-Tuning by Reducing Representational Collapse](https://mlsanity.com/paper/2008.03156/) (2020) - [Multilingual is not enough: BERT for Finnish](https://mlsanity.com/paper/1912.07076/) (2019) - [BERTje: A Dutch BERT Model](https://mlsanity.com/paper/1912.09582/) (2019) - [Automated Concatenation of Embeddings for Structured Prediction](https://mlsanity.com/paper/2010.05006/) (2020) - [What the [MASK]? Making Sense of Language-Specific BERT Models](https://mlsanity.com/paper/2003.02912/) (2020) - [A Corpus for Multilingual Document Classification in Eight Languages](https://mlsanity.com/paper/1805.09821/) (2018) - [One Epoch Is All You Need](https://mlsanity.com/paper/1906.06669/) (2019) - [MarIA: Spanish Language Models](https://mlsanity.com/paper/2107.07253/) (2021) - [BERTIN: Efficient Pre-Training of a Spanish Language Model using Perplexity Sampling](https://mlsanity.com/paper/2207.06814/) (2022) - [Give your Text Representation Models some Love: the Case for Basque](https://mlsanity.com/paper/2004.00033/) (2020) - [KinyaBERT: a Morphology-aware Kinyarwanda Language Model](https://mlsanity.com/paper/2203.08459/) (2022) - [Are Multilingual Models the Best Choice for Moderately Under-resourced Languages? A Comprehensive Assessment for Catalan](https://mlsanity.com/paper/2107.07903/) (2021) - [Allocating Large Vocabulary Capacity for Cross-lingual Language Model Pre-training](https://mlsanity.com/paper/2109.07306/) (2021) - [EstBERT: A Pretrained Language-Specific BERT for Estonian](https://mlsanity.com/paper/2011.04784/) (2020) - [Does Corpus Quality Really Matter for Low-Resource Languages?](https://mlsanity.com/paper/2203.08111/) (2022) ## Built on by (most cited first, 8 of 8) - [Medical mT5: An Open-Source Multilingual Text-to-Text LLM for The Medical Domain](https://mlsanity.com/paper/2404.07613/) (2024) - [Explanatory Argument Extraction of Correct Answers in Resident Medical Exams](https://mlsanity.com/paper/2312.00567/) (2023) - [Cross-lingual Argument Mining in the Medical Domain](https://mlsanity.com/paper/2301.10527/) (2023) - [Sequence-to-Sequence Spanish Pre-trained Language Models](https://mlsanity.com/paper/2309.11259/) (2023) - [Evaluating Shortest Edit Script Methods for Contextual Lemmatization](https://mlsanity.com/paper/2403.16968/) (2024) - [LyricSIM: A novel Dataset and Benchmark for Similarity Detection in Spanish Song LyricS](https://mlsanity.com/paper/2306.01325/) (2023) - [A Survey of Spanish Clinical Language Models](https://mlsanity.com/paper/2308.02199/) (2023) - [MEL: Legal Spanish Language Model](https://mlsanity.com/paper/2501.16011/) (2025) ## Abstract (arXiv) Given the impact of language models on the field of Natural Language Processing, a number of Spanish encoder-only masked language models (aka BERTs) have been trained and released. These models were developed either within large projects using very large private corpora or by means of smaller scale academic efforts leveraging freely available data. In this paper we present a comprehensive head-to-head comparison of language models for Spanish with the following results: (i) Previously ignored multilingual models from large companies fare better than monolingual models, substantially changing the evaluation landscape of language models in Spanish; (ii) Results across the monolingual models are not conclusive, with supposedly smaller and inferior models performing competitively. Based on these empirical results, we argue for the need of more research to understand the factors underlying them. In this sense, the effect of corpus size, quality and pre-training techniques need to be further investigated to be able to obtain Spanish monolingual models significantly better than the multilingual ones released by large private companies, specially in the face of rapid ongoing progress in the field. The recent activity in the development of language technology for Spanish is to be welcomed, but our results show that building language models remains an open, resource-heavy problem which requires to marry resources (monetary and/or computational) with the best research expertise and practice.