mlsanityData to 14 Oct 2026

FineWeb2: One Pipeline to Scale Them All -- Adapting Pre-Training Data Processing to Every Language

arXiv 2506.20920 ↗First posted 26 Jun 2025cs.CLcited by 143

Not on the map. Why?

Abstract (arXiv)

Pre-training state-of-the-art large language models (LLMs) requires vast amounts of clean and diverse text data. While the open development of large high-quality English pre-training datasets has seen substantial recent progress, training performant multilingual LLMs remains a challenge, in large part due to the inherent difficulty of tailoring filtering and deduplication pipelines to a large number of languages. In this work, we introduce a new pre-training dataset curation pipeline based on FineWeb that can be automatically adapted to support any language. We extensively ablate our pipeline design choices on a set of nine diverse languages, guided by a set of meaningful and informative evaluation tasks that were chosen through a novel selection process based on measurable criteria. Ultimately, we show that our pipeline can be used to create non-English corpora that produce more performant models than prior datasets. We additionally introduce a straightforward and principled approach to rebalance datasets that takes into consideration both duplication count and quality, providing an additional performance uplift. Finally, we scale our pipeline to over 1000 languages using almost 100 Common Crawl snapshots to produce FineWeb2, a new 20 terabyte (5 billion document) multilingual dataset which we release along with our pipeline, training, and evaluation codebases.

Key facts. Cited by 143 papers in the map (data through 14 Oct 2026). First posted on arXiv 26 Jun 2025.

Builds on

0 of 0 references.

No references here yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 37; 2026: 106.

143 papers.

Sort:

By area:

  1. 2025GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation Modelscited by 467
  2. 2025NVIDIA Nemotron Nano 2: An Accurate and Efficient Hybrid Mamba-Transformer Reasoning Modelcited by 108
  3. 2025mmBERT: A Modern Multilingual Encoder with Annealed Language Learningcited by 45
  4. 2025Parity-Aware Byte-Pair Encoding: Improving Cross-lingual Fairness in Tokenizationcited by 20
  5. 2026Tiny Aya: Bridging Scale and Multilingual Depthcited by 15
  6. 2025Confounding Factors in Relating Model Performance to Morphologycited by 14
  7. 2025How Learning Rate Decay Wastes Your Best Data in Curriculum-Based LLM Pretrainingcited by 14
  8. 2026AfriqueLLM: How Data Mixing and Model Architecture Impact Continued Pre-training for African Languagescited by 12
  9. 2025Revisiting Multilingual Data Mixtures in Language Model Pretrainingcited by 10
  10. 2026Diffusion-Pretrained Dense and Contextual Embeddingscited by 9
  11. 2026The Finetuner's Fallacy: When to Pretrain with Your Finetuning Datacited by 9
  12. 2025Optimal Scaling Needs Optimal Normcited by 8
38 more
  1. 2025Tracing Multilingual Representations in LLMs with Cross-Layer Transcoderscited by 8
  2. 2025Nile-Chat: Egyptian Language Models for Arabic and Latin Scriptscited by 7
  3. 2025From KMMLU-Redux to KMMLU-Pro: A Professional Korean Benchmark Suite for LLM Evaluationcited by 6
  4. 2025Ibom NLP: A Step Toward Inclusive Natural Language Processing for Nigeria's Minority Languagescited by 6
  5. 2025Bielik 11B v3: Multilingual Large Language Model for European Languagescited by 6
  6. 2026A Survey on Evaluating Quality and Trustworthiness in LLM-Generated Datacited by 6
  7. 2026CommonLID: Re-evaluating State-of-the-Art Language Identification Performance on Web Datacited by 6
  8. 2026propella-1: Multi-Property Document Annotation for LLM Data Curation at Scalecited by 6
  9. 2026Scaling Laws for Mixture Pretraining Under Data Constraintscited by 6
  10. 2026Ling and Ring 2.6 Technical Report: Efficient and Instant Agentic Intelligence at Trillion-Parameter Scalecited by 6
  11. 2025MuRating: A High Quality Data Selecting Approach to Multilingual Large Language Model Pretrainingcited by 5
  12. 2026OpenLID-v3: Improving the Precision of Closely Related Language Identification -- An Experience Reportcited by 5
  13. 2025From Curated Data to Scalable Models: Continual Pre-training of Dense and MoE Large Language Models for Tibetancited by 4
  14. 2025Translate, then Detect: Leveraging Machine Translation for Cross-Lingual Toxicity Classificationcited by 4
  15. 2025Motif 2 12.7B technical reportcited by 4
  16. 2025TokSuite: Measuring the Impact of Tokenizer Choice on Language Model Behaviorcited by 4
  17. 2026Unveiling Language Routing Isolation in Multilingual MoE Models for Interpretable Subnetwork Adaptationcited by 4
  18. 2026Compute Optimal Tokenizationcited by 4
  19. 2025The German Commons - 154 Billion Tokens of Openly Licensed Text for German Language Modelscited by 3
  20. 2025IndicParam: Benchmark to evaluate LLMs on low-resource Indic Languagescited by 3
  21. 2025FiNERweb: Datasets and Artifacts for Scalable Multilingual Named Entity Recognitioncited by 3
  22. 2026Preferences for Idiomatic Language are Acquired Slowly -- and Forgotten Quickly: A Case Study on Swedishcited by 3
  23. 2026"UberWeb: Insights from Multilingual Curation for a 20-Trillion-Token Datasetcited by 3
  24. 2026ShapleyLaw: A Game-Theoretic Approach to Multilingual Scaling Lawscited by 3
  25. 2026JAMMEval: A Refined Collection of Japanese Benchmarks for Reliable VLM Evaluationcited by 3
  26. 2026Safety is Contextual, LLM-Judges Are Not: Navigating the Rigid Priors of Evaluatorscited by 3
  27. 2026Pangram 4 Technical Reportcited by 3
  28. 2025AI Diffusion in Low Resource Language Countriescited by 2
  29. 2025Teaching Old Tokenizers New Words: Efficient Tokenizer Adaptation for Pre-trained Modelscited by 2
  30. 2026Solar Open Technical Reportcited by 2
  31. 2026Mecellem Models: Turkish Models Trained from Scratch and Continually Pre-trained for the Legal Domaincited by 2
  32. 2026Decouple Searching from Training: Scaling Data Mixing via Model Merging for Large Language Model Pre-trainingcited by 2
  33. 2026DataChef: Cooking Up Optimal Data Recipes for LLM Adaptation via Reinforcement Learningcited by 2
  34. 2026TildeOpen LLM: Leveraging Curriculum Learning to Achieve Equitable Language Representationcited by 2
  35. 2026Translation Asymmetry in LLMs as a Data Augmentation Factor: A Case Study for 6 Romansh Language Varietiescited by 2
  36. 2026GlotOCR Bench: OCR Models Still Struggle Beyond a Handful of Unicode Scriptscited by 2
  37. 2026Soro: A Lightweight Foundation Model and Chatbot for Tajikcited by 2
  38. 2026Grouped Query Experts: Mixture-of-Experts on GQA Self-Attentioncited by 2

50 most cited of 143 All 143 (JSON)