mlsanityData to 7 Oct 2026

A Diversity Diet for a Healthier Model: A Case Study of French ModernBERT

arXiv 2602.22014 ↗First posted 25 Feb 2026cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Diversity has been gaining interest in the NLP community in recent years. At the same time, state-of-the-art transformer models such as ModernBERT use very large pre-training datasets, which are driven by size rather than by diversity. This summons for an investigation of the impact of diversity on the ModernBERT pre-training. We do so in this study, with the express intent of reducing pre-training dataset size, while retaining at least comparable performance. We compare diversity-driven sampling algorithms, so as to pick the best one. We find that diversity-driven sampling allows in some tasks to gain 10 points relative to randomly-sampled pre-training data of commensurate size. We also see that a model pre-trained for 483h on a diversity-driven dataset of 150M tokens can yield a commensurate performance to a model pre-trained for 1,775h on a randomly-driven dataset of 2.4B tokens.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 43 of its 87 references that are in the map. First posted on arXiv 25 Feb 2026.

Builds on

43 of 87 references.

References · 43

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Scaling Laws for Neural Language Modelscited by 6,411
  4. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  5. 2020GLU Variants Improve Transformercited by 1,724
  6. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,242
  7. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,104
  8. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 913
  9. 2024Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inferencecited by 527
  10. 2020Universal Dependencies v2: An Evergrowing Multilingual Treebank Collectioncited by 269
  11. 2019PAWS-X: A Cross-lingual Adversarial Dataset for Paraphrase Identificationcited by 262
  12. 2022DeepCore: A Comprehensive Library for Coreset Selection in Deep Learningcited by 146
31 more
  1. 2020Trading Off Diversity and Quality in Natural Language Generationcited by 94
  2. 2024A New Massive Multilingual Dataset for High-Performance Language Technologiescited by 56
  3. 2018Diversity in Machine Learningcited by 48
  4. 2020Generating Diverse and Consistent QA pairs from Contexts with Information-Maximizing Hierarchical Conditional VAEscited by 31
  5. 2021Finding needles in a haystack: Sampling Structurally-diverse Training Sets from Synthetic Data for Compositional Generalizationcited by 27
  6. 2024Scaling Data Diversity for Fine-Tuning Language Models in Human Alignmentcited by 25
  7. 2025Measuring Data Diversity for Instruction Tuning: A Systematic Analysis and A Reliable Metriccited by 24
  8. 2019Exact sampling of determinantal point processes with sublinear time preprocessingcited by 19
  9. 2023NLU on Data Diets: Dynamic Data Subset Selection for NLP Classification Taskscited by 19
  10. 2023JWSign: A Highly Multilingual Corpus of Bible Translations for more Diversity in Sign Language Processingcited by 18
  11. 2022Pyramid-BERT: Reducing Complexity via Successive Core-set based Token Selectioncited by 13
  12. 2024Diverse Perspectives, Divergent Models: Cross-Cultural Evaluation of Depression Detection on Twittercited by 12
  13. 2021Diverse Distributions of Self-Supervised Tasks for Meta-Learning in NLPcited by 10
  14. 2022DeepGen: Diverse Search Ad Generation and Real-Time Customizationcited by 10
  15. 2023Large-Scale and Multi-Perspective Opinion Summarization with Diverse Review Subsetscited by 10
  16. 2024A Measure for Transparent Comparison of Linguistic Diversity in Multilingual NLP Data Setscited by 10
  17. 2021Diversity-Aware Batch Active Learning for Dependency Parsingcited by 9
  18. 2020Deep Active Learning for Sequence Labeling Based on Diversity and Uncertainty in Gradientcited by 8
  19. 2022Language Diversity: Visible to Humans, Exploitable by Machinescited by 8
  20. 2022Structurally Diverse Sampling for Sample-Efficient Training and Comprehensive Evaluationcited by 8
  21. 2022MINION: a Large-Scale and Diverse Dataset for Multilingual Event Detectioncited by 8
  22. 2022Diverse Parallel Data Synthesis for Cross-Database Adaptation of Text-to-SQL Parserscited by 7
  23. 2023Prefix-diffusion: A Lightweight Diffusion Model for Diverse Image Captioningcited by 7
  24. 2024A diverse Multilingual News Headlines Dataset from around the Worldcited by 6
  25. 2020Measuring Diversity in Heterogeneous Information Networkscited by 5
  26. 2022Exploring Diversity in Back Translation for Low-Resource Machine Translationcited by 5
  27. 2022Lazy and Fast Greedy MAP Inference for Determinantal Point Processcited by 5
  28. 2022"Diversity and Uncertainty in Moderation" are the Key to Data Selection for Multilingual Few-shot Transfercited by 5
  29. 2021Lingxi: A Diversity-aware Chinese Modern Poetry Generation Systemcited by 4
  30. 2025A survey of diversity quantification in natural language processing: The why, what, where and howcited by 4
  31. 2020Informed Sampling for Diversity in Concept-to-Text NLGcited by 3

Built on by

Not cited yet.