mlsanityData to 14 Oct 2026

BERTIN: Efficient Pre-Training of a Spanish Language Model using Perplexity Sampling

arXiv 2207.06814 ↗First posted 14 Jul 2022cs.CL cs.AIcited by 30

Not on the map. Why?

Abstract (arXiv)

The pre-training of large language models usually requires massive amounts of resources, both in terms of computation and data. Frequently used web sources such as Common Crawl might contain enough noise to make this pre-training sub-optimal. In this work, we experiment with different sampling methods from the Spanish version of mC4, and present a novel data-centric technique which we name textitperplexity sampling that enables the pre-training of language models in roughly half the amount of steps and using one fifth of the data. The resulting models are comparable to the current state-of-the-art, and even achieve better results for certain tasks. Our work is proof of the versatility of Transformers, and paves the way for small teams to train their models on a limited budget. Our models are available at this hrefhttps://huggingface.co/bertin-projectURL.

Key facts. Cited by 30 papers in the map (data through 14 Oct 2026). Builds on 18 of its 57 references that are in the map. First posted on arXiv 14 Jul 2022.

Builds on

18 of 57 references.

References · 18

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  6. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  7. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  8. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  9. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,818
  10. 2020Language (Technology) is Power: A Critical Survey of "Bias" in NLPcited by 956
  11. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 915
  12. 2021Deduplicating Training Data Makes Language Models Bettercited by 714
6 more
  1. 2019CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Datacited by 548
  2. 2021Pre-Trained Models: Past, Present and Futurecited by 319
  3. 2017Unsupervised Machine Translation Using Monolingual Corpora Onlycited by 273
  4. 2019PAWS-X: A Cross-lingual Adversarial Dataset for Paraphrase Identificationcited by 264
  5. 2019Fair is Better than Sensational:Man is to Doctor as Woman is to Doctorcited by 27
  6. 2021Measuring Model Biases in the Absence of Ground Truthcited by 24

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 4; 2023: 10; 2024: 10; 2025: 3; 2026: 3.

30 papers.

Sort:

By area:

  1. 2023The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Datasetcited by 148
  2. 2023LEXTREME: A Multi-Lingual and Multi-Task Benchmark for the Legal Domaincited by 68
  3. 2023Multilingual Language Models are not Multicultural: A Case Study in Emotioncited by 54
  4. 2023Generative AI Text Classification using Ensemble LLM Approachescited by 29
  5. 2022Measuring Datacited by 18
  6. 2023BLUEX: A benchmark based on Brazilian Leading Universities Entrance eXamscited by 15
  7. 2024I Learn Better If You Speak My Language: Understanding the Superior Performance of Fine-Tuning Large Language Models with LLM-Generated Responsescited by 15
  8. 2023Findings of the VarDial Evaluation Campaign 2023cited by 14
  9. 2022Assessing the impact of contextual information in hate speech detectioncited by 10
  10. 2022Lessons learned from the evaluation of Spanish Language Modelscited by 8
  11. 2024The Impact of Copyrighted Material on Large Language Models: A Norwegian Perspectivecited by 8
  12. 2024NLAS-multi: A Multilingual Corpus of Automatically Generated Natural Language Argumentation Schemescited by 5
18 more
  1. 2023Two-stage Pipeline for Multilingual Dialect Detectioncited by 3
  2. 2024CoLLEGe: Concept Embedding Generation for Large Language Modelscited by 3
  3. 2024Annotation-Efficient Language Model Alignment via Diverse and Representative Response Textscited by 3
  4. 2024What distinguishes conspiracy from critical narratives? A computational analysis of oppositional discoursecited by 3
  5. 2023Sequence-to-Sequence Spanish Pre-trained Language Modelscited by 2
  6. 2023Comparing Styles across Languages: A Cross-Cultural Exploration of Politenesscited by 2
  7. 2024Bridging Linguistic Structure and Mechanistic Interpretability for Conceptual Interpretation in Language Modelscited by 2
  8. 2025La Leaderboard: A Large Language Model Leaderboard for Spanish Varieties and Languages of Spain and Latin Americacited by 2
  9. 2026IDIOLEX: Unified and Continuous Representations for Idiolectal and Stylistic Variationcited by 2
  10. 2022A Spanish dataset for Targeted Sentiment Analysis of political headlinescited by 1
  11. 2024Data Quality Control in Federated Instruction-tuning of Large Language Modelscited by 1
  12. 2025Efficient Response Generation Strategy Selection for Fine-Tuning Large Language Models Through Self-Aligned Perplexitycited by 1
  13. 2023LyricSIM: A novel Dataset and Benchmark for Similarity Detection in Spanish Song LyricScited by 0
  14. 2024Detecci'on Autom'atica de Patolog'ias en Notas Cl'inicas en Espa~nol Combinando Modelos de Lenguaje y Ontolog'ias M'edicoscited by 0
  15. 2024Automatic detection of diseases in Spanish clinical notes combining medical language models and ontologiescited by 0
  16. 2025BOE-XSUM: Extreme Summarization in Clear Language of Spanish Legal Decrees and Notificationscited by 0
  17. 2026Data Difficulty and the Generalization--Extrapolation Tradeoff in LLM Fine-Tuningcited by 0
  18. 2026From Exposure to Expectation: Frequency, Surprisal, and Language Across Development in Spanishcited by 0