mlsanityData to 14 Oct 2026

Spanish Pre-trained BERT Model and Evaluation Data

arXiv 2308.02976 ↗First posted 6 Aug 2023cs.CL cs.AI cs.LGcited by 81

Not on the map. Why?

Abstract (arXiv)

The Spanish language is one of the top 5 spoken languages in the world. Nevertheless, finding resources to train or evaluate Spanish language models is not an easy task. In this paper we help bridge this gap by presenting a BERT-based language model pre-trained exclusively on Spanish data. As a second contribution, we also compiled several tasks specifically for the Spanish language in a single repository much in the spirit of the GLUE benchmark. By fine-tuning our pre-trained Spanish model, we obtain better results compared to other BERT-based models pre-trained on multilingual corpora for most of the tasks, even achieving a new state-of-the-art on some of them. We have publicly released our model, the pre-training data, and the compilation of the Spanish benchmarks.

Key facts. Cited by 81 papers in the map (data through 14 Oct 2026). Builds on 32 of its 50 references that are in the map. First posted on arXiv 6 Aug 2023.

Builds on

32 of 50 references.

References · 32

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  5. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  6. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  7. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  8. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  9. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,811
  10. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,790
  11. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682
  12. 2018Deep contextualized word representationscited by 2,290
20 more
  1. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,989
  2. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,475
  3. 2019Cross-lingual Language Model Pretrainingcited by 1,090
  4. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 915
  5. 2019SpanBERT: Improving Pre-training by Representing and Predicting Spanscited by 834
  6. 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 674
  7. 2019On the Cross-lingual Transferability of Monolingual Representationscited by 578
  8. 2019PAWS: Paraphrase Adversaries from Word Scramblingcited by 402
  9. 2019CamemBERT: a Tasty French Language Modelcited by 368
  10. 2019MLQA: Evaluating Cross-lingual Extractive Question Answeringcited by 340
  11. 2019Beto, Bentz, Becas: The Surprising Cross-Lingual Effectiveness of BERTcited by 301
  12. 2019PAWS-X: A Cross-lingual Adversarial Dataset for Paraphrase Identificationcited by 264
  13. 2019FlauBERT: Unsupervised Language Model Pre-training for Frenchcited by 153
  14. 2019Multilingual is not enough: BERT for Finnishcited by 119
  15. 2019Adaptation of Deep Bidirectional Multilingual Transformers for Russian Languagecited by 109
  16. 2019BERTje: A Dutch BERT Modelcited by 109
  17. 2020RobBERT: a Dutch RoBERTa-based Language Modelcited by 88
  18. 2019Portuguese Named Entity Recognition using BERT-CRFcited by 73
  19. 2018A Corpus for Multilingual Document Classification in Eight Languagescited by 42
  20. 2019Automatic Spanish Translation of the SQuAD Dataset for Multilingual Question Answeringcited by 35

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 14; 2024: 38; 2025: 13; 2026: 16.

81 papers.

Sort:

By area:

  1. 2024INCLUDE: Evaluating Multilingual Language Understanding with Regional Knowledgecited by 86
  2. 2024SemRel2024: A Collection of Semantic Textual Relatedness Datasets for 13 Languagescited by 16
  3. 2023Understanding writing style in social media with a supervised contrastively pre-trained transformercited by 12
  4. 2024Few-shot clinical entity recognition in English, French and Spanish: masked language models outperform generative model promptingcited by 12
  5. 2023Explanatory Argument Extraction of Correct Answers in Resident Medical Examscited by 6
  6. 2024Comparing Explanation Faithfulness between Multilingual and Monolingual Fine-tuned Language Modelscited by 6
  7. 2024A Survey of Large Language Models for European Languagescited by 6
  8. 2023Early Detection of Depression and Eating Disorders in Spanish: UNSL at MentalRiskES 2023cited by 5
  9. 2024Credit Risk Meets Large Language Models: Building a Risk Indicator from Loan Descriptions in P2P Lendingcited by 5
  10. 2024Different Tokenization Schemes Lead to Comparable Performance in Spanish Number Agreementcited by 5
  11. 2024Evaluating Contextualized Representations of (Spanish) Ambiguous Words: A New Lexical Resource and Empirical Analysiscited by 5
  12. 2024Is It Good Data for Multilingual Instruction Tuning or Just Bad Multilingual Evaluation for Large Language Models?cited by 4
38 more
  1. 2025Examining Spanish Counseling with MIDAS: a Motivational Interviewing Dataset in Spanishcited by 4
  2. 2025Overview of ADoBo at IberLEF 2025: Automatic Detection of Anglicisms in Spanishcited by 4
  3. 2024LLMs for Targeted Sentiment in News Headlines: Exploring the Descriptive-Prescriptive Dilemmacited by 3
  4. 2024Zero-Shot Cross-Lingual Document-Level Event Causality Identification with Heterogeneous Graph Contrastive Transfer Learningcited by 3
  5. 2024What distinguishes conspiracy from critical narratives? A computational analysis of oppositional discoursecited by 3
  6. 2024Common Ground, Diverse Roots: The Difficulty of Classifying Common Examples in Spanish Varietiescited by 3
  7. 2026VectraYX-Nano: A 42M-Parameter Spanish Cybersecurity Language Model with Curriculum Learning and Native Tool Usecited by 3
  8. 2023Analysing Cross-Lingual Transfer in Low-Resourced African Named Entity Recognitioncited by 2
  9. 2023Sequence-to-Sequence Spanish Pre-trained Language Modelscited by 2
  10. 2023Is Probing All You Need? Indicator Tasks as an Alternative to Probing Embedding Spacescited by 2
  11. 2024Cascaded Cross-Modal Transformer for Audio-Textual Classificationcited by 2
  12. 2024What is Your Favorite Gender, MLM? Gender Bias Evaluation in Multilingual Masked Language Modelscited by 2
  13. 2024Unknown Script: Impact of Script on Cross-Lingual Transfercited by 2
  14. 2024Historical Ink: 19th Century Latin American Spanish Newspaper Corpus with LLM OCR Correctioncited by 2
  15. 2024The #Somos600M Project: Generating NLP resources that represent the diversity of the languages from LATAM, the Caribbean, and Spaincited by 2
  16. 2024The Future of Intelligent Healthcare: A Systematic Analysis and Discussion on the Integration and Impact of Robots Using Large Language Models for Healthcarecited by 2
  17. 2025Culinary Crossroads: A RAG Framework for Enhancing Diversity in Cross-Cultural Recipe Adaptationcited by 2
  18. 2023A User-Centered Evaluation of Spanish Text Simplificationcited by 1
  19. 2023Knowledge Graph Embeddings for Multi-Lingual Structured Representations of Radiology Reportscited by 1
  20. 2023Transformer-based Entity Legal Form Classificationcited by 1
  21. 2023Deep Natural Language Feature Learning for Interpretable Predictioncited by 1
  22. 2023A Novel Dataset for Financial Education Text Simplification in Spanishcited by 1
  23. 2024Pointer-Generator Networks for Low-Resource Machine Translation: Don't Copy That!cited by 1
  24. 2024MasonTigers at SemEval-2024 Task 1: An Ensemble Approach for Semantic Textual Relatednesscited by 1
  25. 2024Evaluating Shortest Edit Script Methods for Contextual Lemmatizationcited by 1
  26. 2024Sentiment Analysis Across Languages: Evaluation Before and After Machine Translation to Englishcited by 1
  27. 2024Mining Reasons For And Against Vaccination From Unstructured Data Using Nichesourcing and AI Data Augmentationcited by 1
  28. 2024Historical Ink: Semantic Shift Detection for 19th Century Spanishcited by 1
  29. 2024Exploring Large Language Models for Hate Speech Detection in Rioplatense Spanishcited by 1
  30. 2024VL-GLUE: A Suite of Fundamental yet Challenging Visuo-Linguistic Reasoning Taskscited by 1
  31. 2024A Time-Aware Approach to Early Detection of Anorexia: UNSL at eRisk 2024cited by 1
  32. 2024Deep Insights into Cognitive Decline: A Survey of Leveraging Non-Intrusive Modalities with Deep Learning Techniquescited by 1
  33. 2025Predicting potentially abusive clauses in Chilean terms of services with natural language processingcited by 1
  34. 2025Temporal fine-tuning for early risk detectioncited by 1
  35. 2025SLIM-LLMs: Modeling of Style-Sensory Language RelationshipsThrough Low-Dimensional Representationscited by 1
  36. 2025Multilingual Clinical NER for Diseases and Medications Recognition in Cardiology Texts using BERT Embeddingscited by 1
  37. 2026Scalable and Culturally Specific Stereotype Dataset Construction via Human-LLM Collaborationcited by 1
  38. 2023Unsupervised Lexical Simplification with Context Augmentationcited by 0

50 most cited of 81 All 81 (JSON)