mlsanityData to 7 Oct 2026

Organize the Web: Constructing Domains Enhances Pre-Training Data Curation

arXiv 2502.10341 ↗First posted 14 Feb 2025cs.CLcited by 73

Not on the map. Why?

Abstract (arXiv)

Modern language models are trained on large, unstructured datasets consisting of trillions of tokens and obtained by crawling the web. The unstructured nature makes it difficult to reason about their contents and develop systematic approaches to data curation. In this paper, we unpack monolithic web corpora by developing taxonomies of their contents and organizing them into domains. We introduce WebOrganizer, a framework for organizing web pages in terms of both their topic and format. Using these two complementary notions of domains, we automatically annotate pre-training data by distilling annotations from a large language model into efficient classifiers. This allows us to study how data from different domains should be mixed to improve models on downstream tasks, and we show that we can combine insights about effective topics and formats to further boost performance. We demonstrate that our domain mixing also improves existing methods that select data based on quality. Furthermore, we study and compare how quality-based methods will implicitly change the domain mixture. Overall, our work demonstrates that constructing and mixing domains provides a valuable complement to quality-based data curation methods, opening new avenues for effective and insightful pre-training data curation.

Key facts. Cited by 73 papers in the map (data through 7 Oct 2026). Builds on 70 of its 88 references that are in the map. First posted on arXiv 14 Feb 2025.

Builds on

70 of 88 references.

References · 70

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  7. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  8. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  9. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  10. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  11. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  12. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
38 more
  1. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  2. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
  3. 2020GLU Variants Improve Transformercited by 1,724
  4. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  5. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,107
  6. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,104
  7. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,041
  8. 2016Bag of Tricks for Efficient Text Classificationcited by 869
  9. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 825
  10. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 711
  11. 2021Deduplicating Training Data Makes Language Models Bettercited by 705
  12. 2023Towards General Text Embeddings with Multi-stage Contrastive Learningcited by 705
  13. 2019CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Datacited by 544
  14. 2024Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Researchcited by 475
  15. 2023Scaling Data-Constrained Language Modelscited by 391
  16. 2023DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretrainingcited by 385
  17. 2024DataComp-LM: In search of the next generation of training sets for language modelscited by 381
  18. 2023Data Selection for Language Models via Importance Resamplingcited by 322
  19. 20242 OLMo 2 Furiouscited by 302
  20. 2023SemDeDup: Data-efficient learning at web-scale through semantic deduplicationcited by 295
  21. 2023A Pretrainer's Guide to Training Data: Measuring the Effects of Data Age, Domain Coverage, Quality, & Toxicitycited by 180
  22. 2024Data Mixing Laws: Optimizing Data Mixtures by Predicting Language Modeling Performancecited by 179
  23. 2024RegMix: Data Mixture as Regression for Language Model Pre-trainingcited by 172
  24. 2024Rephrasing the Web: A Recipe for Compute and Data-Efficient Language Modelingcited by 156
  25. 2024QuRating: Selecting High-Quality Data for Training Language Modelscited by 153
  26. 2023D4: Improving LLM Pretraining via Document De-Duplication and Diversificationcited by 146
  27. 2023What's In My Big Data?cited by 129
  28. 2023Skill-it! A Data-Driven Skills Framework for Understanding and Training Language Modelscited by 120
  29. 2024DsDm: Model-Aware Dataset Selection with Datamodelscited by 103
  30. 2024OLMES: A Standard for Language Model Evaluationscited by 100
  31. 2024MATES: Model-Aware Data Selection for Efficient Pretraining with Data Influence Modelscited by 96
  32. 2024How to Train Data-Efficient LLMscited by 91
  33. 2023DoGE: Domain Reweighting with Generalization Estimationcited by 87
  34. 2024Perplexed by Perplexity: Perplexity-Based Data Pruning With Small Reference Modelscited by 83
  35. 2023At Which Training Stage Does Code Data Help LLMs Reasoning?cited by 82
  36. 2023Efficient Online Data Mixing For Language Model Pre-Trainingcited by 77
  37. 2023ClusterLLM: Large Language Models as a Guide for Text Clusteringcited by 69
  38. 2024Clio: Privacy-Preserving Insights into Real-World AI Usecited by 66

50 most cited of 70 All 70 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 39; 2026: 34.

73 papers.

Sort:

By area:

  1. 2025Qwen3-VL Technical Reportcited by 2,534
  2. 2025OpenThoughts: Data Recipes for Reasoning Modelscited by 287
  3. 2025DeepResearch Bench: A Comprehensive Benchmark for Deep Research Agentscited by 236
  4. 2025olmOCR: Unlocking Trillions of Tokens in PDFs with Vision Language Modelscited by 120
  5. 2025Nemotron-CLIMB: CLustering-based Iterative Data Mixture Bootstrapping for Language Model Pre-trainingcited by 62
  6. 2025Learning to Reason Across Parallel Samples for LLM Reasoningcited by 32
  7. 2025Recycling the Web: A Method to Enhance Pre-training Data Quality and Quantity for Language Modelscited by 31
  8. 2025Signal and Noise: A Framework for Reducing Uncertainty in Language Model Evaluationcited by 22
  9. 2025BeyondWeb: Lessons from Scaling Synthetic Data for Trillion-scale Pretrainingcited by 19
  10. 2025Language Models Improve When Pretraining Data Matches Target Taskscited by 18
  11. 2026OPUS: Towards Efficient and Principled Data Selection in Large Language Model Pre-training in Every Iterationcited by 17
  12. 2025FlexOlmo: Open Language Models for Flexible Data Usecited by 14
38 more
  1. 2026DeepResearchEval: An Automated Framework for Deep Research Task Construction and Agentic Evaluationcited by 13
  2. 2025Group-Level Data Selection for Efficient Pretrainingcited by 12
  3. 2025How Learning Rate Decay Wastes Your Best Data in Curriculum-Based LLM Pretrainingcited by 12
  4. 2025DEER: A Benchmark for Evaluating Deep Research Agents on Expert Report Generationcited by 12
  5. 2026AfriqueLLM: How Data Mixing and Model Architecture Impact Continued Pre-training for African Languagescited by 12
  6. 2026GradAlign: Gradient-Aligned Data Selection for LLM Reinforcement Learningcited by 10
  7. 2025Why Do Some Inputs Break Low-Bit LLM Quantization?cited by 8
  8. 2025Gaperon: A Peppered English-French Generative Language Model Suitecited by 8
  9. 2025Dripper: Token-Efficient Main HTML Extraction with a Lightweight LMcited by 8
  10. 2025LLM Fingerprinting via Semantically Conditioned Watermarkscited by 7
  11. 2025Essential-Web v1.0: 24T tokens of organized web datacited by 6
  12. 2025TiKMiX: Take Data Influence into Dynamic Mixture for Language Model Pre-trainingcited by 6
  13. 2025AttentionInfluence: Adopting Attention Head Influence for Weak-to-Strong Pretraining Data Selectioncited by 5
  14. 2025Data Mixing Agent: Learning to Re-weight Domains for Continual Pre-trainingcited by 5
  15. 2025Topic Over Source: The Key to Effective Data Mixing for Language Models Pre-trainingcited by 4
  16. 2025Semantic Probabilistic Control of Language Modelscited by 4
  17. 2025OLMoASR: Open Models and Data for Training Robust Speech Recognition Modelscited by 4
  18. 2025so much depends / upon / a whitespace: Why Whitespace Matters for Poets and LLMscited by 4
  19. 2025Joint Selection for Large-Scale Pre-Training Data via Policy Gradient-based Mask Learningcited by 4
  20. 2025R&B: Domain Regrouping and Data Mixture Balancing for Efficient Foundation Model Trainingcited by 3
  21. 2026Suppressing Final Layer Hidden State Jumps in Transformer Pretrainingcited by 3
  22. 2026Expert Divergence Learning for MoE-based Language Modelscited by 3
  23. 2026Evaluating Progress in Graph Foundation Models: A Comprehensive Benchmark and New Insightscited by 3
  24. 2026Data Darwinism Part II: DataEvolve -- AI can Autonomously Evolve Pretraining Data Curationcited by 3
  25. 2026EMO: Pretraining Mixture of Experts for Emergent Modularitycited by 3
  26. 2025Can Performant LLMs Be Ethical? Quantifying the Impact of Web Crawling Opt-Outscited by 2
  27. 2025Biomed-Enriched: A Biomedical Dataset Enriched with LLMs for Pretraining and Extracting Rare and Hidden Contentcited by 2
  28. 2025Data-Centric Lessons To Improve Speech-Language Pretrainingcited by 2
  29. 2025AIR: Post-training Data Selection for Reasoning via Attention Head Influencecited by 2
  30. 2026Mecellem Models: Turkish Models Trained from Scratch and Continually Pre-trained for the Legal Domaincited by 2
  31. 2026Data Science and Technology Towards AGI Part I: Tiered Data Managementcited by 2
  32. 2026How Does "English (US)" Become the Default? Triangulating Structural Bias Towards American English Across the LLM Pipelinecited by 2
  33. 2026MixAtlas: Uncertainty-aware Data Mixture Optimization for Multimodal LLM Midtrainingcited by 2
  34. 2025URLs Help, Topics Guide: Understanding Metadata Utility in LLM Trainingcited by 1
  35. 2025DoTA-RAG: Dynamic of Thought Aggregation RAGcited by 1
  36. 2025Interactive Training: Feedback-Driven Neural Network Optimizationcited by 1
  37. 2025PCMind-2.1-Kaiyuan-2B Technical Reportcited by 1
  38. 2026Neural Neural Scaling Lawscited by 1

50 most cited of 73 All 73 (JSON)