# BERTIN: Efficient Pre-Training of a Spanish Language Model using Perplexity Sampling

arXiv: https://arxiv.org/abs/2207.06814 · HTML: https://mlsanity.com/paper/2207.06814/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 57 references that are in the map.
- First posted on arXiv 14 Jul 2022.

## Builds on (most cited first, 18 of 18)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity](https://mlsanity.com/paper/2101.03961/) (2021)
- [The Pile: An 800GB Dataset of Diverse Text for Language Modeling](https://mlsanity.com/paper/2101.00027/) (2020)
- [mT5: A massively multilingual pre-trained text-to-text transformer](https://mlsanity.com/paper/2010.11934/) (2020)
- [Language (Technology) is Power: A Critical Survey of "Bias" in NLP](https://mlsanity.com/paper/2005.14050/) (2020)
- [XNLI: Evaluating Cross-lingual Sentence Representations](https://mlsanity.com/paper/1809.05053/) (2018)
- [Deduplicating Training Data Makes Language Models Better](https://mlsanity.com/paper/2107.06499/) (2021)
- [CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Data](https://mlsanity.com/paper/1911.00359/) (2019)
- [Pre-Trained Models: Past, Present and Future](https://mlsanity.com/paper/2106.07139/) (2021)
- [Unsupervised Machine Translation Using Monolingual Corpora Only](https://mlsanity.com/paper/1711.00043/) (2017)
- [PAWS-X: A Cross-lingual Adversarial Dataset for Paraphrase Identification](https://mlsanity.com/paper/1908.11828/) (2019)
- [Fair is Better than Sensational:Man is to Doctor as Woman is to Doctor](https://mlsanity.com/paper/1905.09866/) (2019)
- [Measuring Model Biases in the Absence of Ground Truth](https://mlsanity.com/paper/2103.03417/) (2021)

## Built on by (most cited first, 30 of 30)

- [The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Dataset](https://mlsanity.com/paper/2303.03915/) (2023)
- [LEXTREME: A Multi-Lingual and Multi-Task Benchmark for the Legal Domain](https://mlsanity.com/paper/2301.13126/) (2023)
- [Multilingual Language Models are not Multicultural: A Case Study in Emotion](https://mlsanity.com/paper/2307.01370/) (2023)
- [Generative AI Text Classification using Ensemble LLM Approaches](https://mlsanity.com/paper/2309.07755/) (2023)
- [Measuring Data](https://mlsanity.com/paper/2212.05129/) (2022)
- [BLUEX: A benchmark based on Brazilian Leading Universities Entrance eXams](https://mlsanity.com/paper/2307.05410/) (2023)
- [I Learn Better If You Speak My Language: Understanding the Superior Performance of Fine-Tuning Large Language Models with LLM-Generated Responses](https://mlsanity.com/paper/2402.11192/) (2024)
- [Findings of the VarDial Evaluation Campaign 2023](https://mlsanity.com/paper/2305.20080/) (2023)
- [Assessing the impact of contextual information in hate speech detection](https://mlsanity.com/paper/2210.00465/) (2022)
- [Lessons learned from the evaluation of Spanish Language Models](https://mlsanity.com/paper/2212.08390/) (2022)
- [The Impact of Copyrighted Material on Large Language Models: A Norwegian Perspective](https://mlsanity.com/paper/2412.09460/) (2024)
- [NLAS-multi: A Multilingual Corpus of Automatically Generated Natural Language Argumentation Schemes](https://mlsanity.com/paper/2402.14458/) (2024)
- [Two-stage Pipeline for Multilingual Dialect Detection](https://mlsanity.com/paper/2303.03487/) (2023)
- [CoLLEGe: Concept Embedding Generation for Large Language Models](https://mlsanity.com/paper/2403.15362/) (2024)
- [Annotation-Efficient Language Model Alignment via Diverse and Representative Response Texts](https://mlsanity.com/paper/2405.13541/) (2024)
- [What distinguishes conspiracy from critical narratives? A computational analysis of oppositional discourse](https://mlsanity.com/paper/2407.10745/) (2024)
- [Sequence-to-Sequence Spanish Pre-trained Language Models](https://mlsanity.com/paper/2309.11259/) (2023)
- [Comparing Styles across Languages: A Cross-Cultural Exploration of Politeness](https://mlsanity.com/paper/2310.07135/) (2023)
- [Bridging Linguistic Structure and Mechanistic Interpretability for Conceptual Interpretation in Language Models](https://mlsanity.com/paper/2408.11827/) (2024)
- [La Leaderboard: A Large Language Model Leaderboard for Spanish Varieties and Languages of Spain and Latin America](https://mlsanity.com/paper/2507.00999/) (2025)
- [IDIOLEX: Unified and Continuous Representations for Idiolectal and Stylistic Variation](https://mlsanity.com/paper/2604.04704/) (2026)
- [A Spanish dataset for Targeted Sentiment Analysis of political headlines](https://mlsanity.com/paper/2208.13947/) (2022)
- [Data Quality Control in Federated Instruction-tuning of Large Language Models](https://mlsanity.com/paper/2410.11540/) (2024)
- [Efficient Response Generation Strategy Selection for Fine-Tuning Large Language Models Through Self-Aligned Perplexity](https://mlsanity.com/paper/2502.11779/) (2025)
- [LyricSIM: A novel Dataset and Benchmark for Similarity Detection in Spanish Song LyricS](https://mlsanity.com/paper/2306.01325/) (2023)
- [Detecci'on Autom'atica de Patolog'ias en Notas Cl'inicas en Espa~nol Combinando Modelos de Lenguaje y Ontolog'ias M'edicos](https://mlsanity.com/paper/2410.00616/) (2024)
- [Automatic detection of diseases in Spanish clinical notes combining medical language models and ontologies](https://mlsanity.com/paper/2412.03176/) (2024)
- [BOE-XSUM: Extreme Summarization in Clear Language of Spanish Legal Decrees and Notifications](https://mlsanity.com/paper/2509.24908/) (2025)
- [Data Difficulty and the Generalization--Extrapolation Tradeoff in LLM Fine-Tuning](https://mlsanity.com/paper/2605.12906/) (2026)
- [From Exposure to Expectation: Frequency, Surprisal, and Language Across Development in Spanish](https://mlsanity.com/paper/2608.22452/) (2026)

## Abstract (arXiv)

The pre-training of large language models usually requires massive amounts of resources, both in terms of computation and data. Frequently used web sources such as Common Crawl might contain enough noise to make this pre-training sub-optimal. In this work, we experiment with different sampling methods from the Spanish version of mC4, and present a novel data-centric technique which we name textitperplexity sampling that enables the pre-training of language models in roughly half the amount of steps and using one fifth of the data. The resulting models are comparable to the current state-of-the-art, and even achieve better results for certain tasks. Our work is proof of the versatility of Transformers, and paves the way for small teams to train their models on a limited budget. Our models are available at this hrefhttps://huggingface.co/bertin-projectURL.