mlsanityData to 7 Oct 2026

The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale

arXiv 2406.17557 ↗First posted 25 Jun 2024cs.CLcited by 1,104

Not on the map. Why?

Cited by 46 in Mixture-of-experts, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

The performance of a large language model (LLM) depends heavily on the quality and size of its pretraining dataset. However, the pretraining datasets for state-of-the-art open LLMs like Llama 3 and Mixtral are not publicly available and very little is known about how they were created. In this work, we introduce FineWeb, a 15-trillion token dataset derived from 96 Common Crawl snapshots that produces better-performing LLMs than other open pretraining datasets. To advance the understanding of how best to curate high-quality pretraining datasets, we carefully document and ablate all of the design choices used in FineWeb, including in-depth investigations of deduplication and filtering strategies. In addition, we introduce FineWeb-Edu, a 1.3-trillion token collection of educational text filtered from FineWeb. LLMs pretrained on FineWeb-Edu exhibit dramatically better performance on knowledge- and reasoning-intensive benchmarks like MMLU and ARC. Along with our datasets, we publicly release our data curation codebase and all of the models trained during our ablation experiments.

Key facts. Cited by 1,104 papers in the map (data through 7 Oct 2026). Builds on 34 of its 67 references that are in the map. First posted on arXiv 25 Jun 2024.

Builds on

34 of 67 references.

References · 34

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  7. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  8. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  9. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  10. 2018UMAP: Uniform Manifold Approximation and Projection for Dimension Reductioncited by 2,841
  11. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  12. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
22 more
  1. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  2. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
  3. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
  4. 2016Bag of Tricks for Efficient Text Classificationcited by 869
  5. 2022Galactica: A Large Language Model for Sciencecited by 736
  6. 2024Yi: Open Foundation Models by 01.AIcited by 706
  7. 2024Self-Rewarding Language Modelscited by 654
  8. 2023Textbooks Are All You Needcited by 561
  9. 2019The Woman Worked as a Babysitter: On Biases in Language Generationcited by 540
  10. 2024Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Researchcited by 475
  11. 2022Deduplicating Training Data Mitigates Privacy Risks in Language Modelscited by 309
  12. 2019Measuring Bias in Contextualized Word Representationscited by 275
  13. 2023From Pretraining Data to Language Models to Downstream Tasks: Tracking the Trails of Political Biases Leading to Unfair NLP Modelscited by 257
  14. 2018A Simple Method for Commonsense Reasoningcited by 223
  15. 2019Gender Bias in Contextualized Word Embeddingscited by 203
  16. 2023Self-Alignment with Instruction Backtranslationcited by 149
  17. 2023The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Datasetcited by 148
  18. 2022Towards a Cleaner Document-Oriented Multilingual Crawled Corpuscited by 143
  19. 2019Evaluating the Underlying Gender Bias in Contextualized Word Embeddingscited by 94
  20. 2024MAP-Neo: Highly Capable and Transparent Bilingual Large Language Model Seriescited by 67
  21. 2024Arctic-Embed: Scalable, Efficient, and Accurate Text Embedding Modelscited by 60
  22. 2023Paloma: A Benchmark for Evaluating Language Model Fitcited by 53

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 89; 2025: 403; 2026: 612.

1,104 papers.

Sort:

By area:

  1. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMscited by 1,193
  2. 2024Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inferencecited by 527
  3. 2024Gated Delta Networks: Improving Mamba2 with Delta Rulecited by 483
  4. 2024Titans: Learning to Memorize at Test Timecited by 303
  5. 2025GUI-R1 : A Generalist R1-Style Vision-Language Action Model For GUI Agentscited by 240
  6. 2025DiffuCoder: Understanding and Improving Masked Diffusion Models for Code Generationcited by 184
  7. 2025BrowseComp-Plus: A More Fair and Transparent Evaluation Benchmark of Deep-Research Agentcited by 183
  8. 2024RedPajama: an Open Dataset for Training Large Language Modelscited by 178
  9. 2024Building and better understanding vision-language models: insights and future directionscited by 169
  10. 2024Nemotron-CC: Transforming Common Crawl into a Refined Long-Horizon Pretraining Datasetcited by 153
  11. 2025SoTA with Less: MCTS-Guided Sample Selection for Data-Efficient Visual Reasoning Self-Improvementcited by 144
  12. 2024How to Train Long-Context Language Models (Effectively)cited by 141
38 more
  1. 2024OpenCoder: The Open Cookbook for Top-Tier Code Large Language Modelscited by 127
  2. 2025Echo Chamber: RL Post-training Amplifies Behaviors Learned in Pretrainingcited by 123
  3. 2025olmOCR: Unlocking Trillions of Tokens in PDFs with Vision Language Modelscited by 120
  4. 2025Mixture-of-Recursions: Learning Dynamic Recursive Depths for Adaptive Token-Level Computationcited by 120
  5. 2025MiMo: Unlocking the Reasoning Potential of Language Model -- From Pretraining to Posttrainingcited by 117
  6. 2024Small Language Models: Survey, Measurements, and Insightscited by 109
  7. 2025The Polar Express: Optimal Matrix Sign Methods and Their Application to the Muon Algorithmcited by 104
  8. 2024LongLLaVA: Scaling Multi-modal LLMs to 1000 Images Efficiently via a Hybrid Architecturecited by 100
  9. 2024Steering Language Model Refusal with Sparse Autoencoderscited by 89
  10. 2025GUI-Actor: Coordinate-Free Visual Grounding for GUI Agentscited by 89
  11. 2025Dream-Coder 7B: An Open Diffusion Language Model for Codecited by 81
  12. 2024INCLUDE: Evaluating Multilingual Language Understanding with Regional Knowledgecited by 78
  13. 2026Mamba-3: Improved Sequence Modeling using State Space Principlescited by 76
  14. 2025NorMuon: Making Muon more efficient and scalablecited by 74
  15. 2025Organize the Web: Constructing Domains Enhances Pre-Training Data Curationcited by 73
  16. 2025Nested Learning: The Illusion of Deep Learning Architecturescited by 69
  17. 2025Large Video Planner Enables Generalizable Robot Controlcited by 67
  18. 2025Dynamic Chunking for End-to-End Hierarchical Sequence Modelingcited by 65
  19. 2025Dion: Distributed Orthonormalized Updatescited by 64
  20. 2025Nemotron-CLIMB: CLustering-based Iterative Data Mixture Bootstrapping for Language Model Pre-trainingcited by 62
  21. 2025KVLink: Accelerating Large Language Models via Efficient KV Cache Reusecited by 60
  22. 2025It's All Connected: A Journey Through Test-Time Memorization, Attentional Bias, Retention, and Online Optimizationcited by 60
  23. 2025ATLAS: Learning to Optimally Memorize the Context at Test Timecited by 57
  24. 2024On the Diversity of Synthetic Data and its Impact on Training Large Language Modelscited by 56
  25. 2025From Reflection to Perfection: Scaling Inference-Time Optimization for Text-to-Image Diffusion Models via Reflection Tuningcited by 53
  26. 2025How much do language models memorize?cited by 53
  27. 2025M+: Extending MemoryLLM with Scalable Long-Term Memorycited by 51
  28. 2025MiniCPM4: Ultra-Efficient LLMs on End Devicescited by 51
  29. 2024Collapse or Thrive? Perils and Promises of Synthetic Data in a Self-Generating Worldcited by 49
  30. 2025Scaling Laws for Optimal Data Mixturescited by 49
  31. 2026LoopFormer: Elastic-Depth Looped Transformers for Latent Reasoning via Shortcut Modulationcited by 48
  32. 2025MegaMath: Pushing the Limits of Open Math Corporacited by 47
  33. 2025Attention Sinks and Compression Valleys in LLMs are Two Sides of the Same Coincited by 47
  34. 2024Improving Pretraining Data Using Perplexity Correlationscited by 46
  35. 2025Diffusion Language Models are Super Data Learnerscited by 45
  36. 2025MoM: Linear Sequence Modeling with Mixture-of-Memoriescited by 44
  37. 2025Scaling Laws of Synthetic Data for Language Modelscited by 44
  38. 2024Falcon Mamba: The First Competitive Attention-free 7B Language Modelcited by 43

50 most cited of 1,104 All 1,104 (JSON)