mlsanityData to 13 Oct 2026

DataComp-LM: In search of the next generation of training sets for language models

arXiv 2406.11794 ↗First posted 17 Jun 2024cs.LG cs.CLcited by 390

Not on the map. Why?

Cited by 29 in Mixture-of-experts, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We introduce DataComp for Language Models (DCLM), a testbed for controlled dataset experiments with the goal of improving language models. As part of DCLM, we provide a standardized corpus of 240T tokens extracted from Common Crawl, effective pretraining recipes based on the OpenLM framework, and a broad suite of 53 downstream evaluations. Participants in the DCLM benchmark can experiment with data curation strategies such as deduplication, filtering, and data mixing at model scales ranging from 412M to 7B parameters. As a baseline for DCLM, we conduct extensive experiments and find that model-based filtering is key to assembling a high-quality training set. The resulting dataset, DCLM-Baseline enables training a 7B parameter language model from scratch to 64% 5-shot accuracy on MMLU with 2.6T training tokens. Compared to MAP-Neo, the previous state-of-the-art in open-data language models, DCLM-Baseline represents a 6.6 percentage point improvement on MMLU while being trained with 40% less compute. Our baseline model is also comparable to Mistral-7B-v0.3 and Llama 3 8B on MMLU (63% & 66%), and performs similarly on an average of 53 natural language understanding tasks while being trained with 6.6x less compute than Llama 3 8B. Our results highlight the importance of dataset design for training language models and offer a starting point for further research on data curation.

Key facts. Cited by 390 papers in the map (data through 13 Oct 2026). First posted on arXiv 17 Jun 2024.

Builds on

0 of 0 references.

No references here yet.

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2024: 52; 2025: 158; 2026: 180.

390 papers.

Sort:

By area:

  1. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMscited by 1,201
  2. 2026GLM-5: from Vibe Coding to Agentic Engineeringcited by 542
  3. 2025Dream 7B: Diffusion Large Language Modelscited by 528
  4. 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 298
  5. 2025SmolVLM: Redefining small and efficient multimodal modelscited by 233
  6. 2024RedPajama: an Open Dataset for Training Large Language Modelscited by 180
  7. 2024RoboMIND: Benchmark on Multi-embodiment Intelligence Normative Data for Robot Manipulationcited by 168
  8. 2024Nemotron-CC: Transforming Common Crawl into a Refined Long-Horizon Pretraining Datasetcited by 153
  9. 2025UniTok: A Unified Tokenizer for Visual Generation and Understandingcited by 149
  10. 2024How to Train Long-Context Language Models (Effectively)cited by 144
  11. 2025olmOCR: Unlocking Trillions of Tokens in PDFs with Vision Language Modelscited by 123
  12. 2024Scaling Laws with Vocabulary: Larger Models Deserve Larger Vocabulariescited by 116
38 more
  1. 2025OctoThinker: Mid-training Incentivizes Reinforcement Learning Scalingcited by 113
  2. 2024Small Language Models: Survey, Measurements, and Insightscited by 110
  3. 2024On-Device Language Models: A Comprehensive Reviewcited by 102
  4. 2024Scaling Laws for Precisioncited by 94
  5. 2025A Sober Look at Progress in Language Model Reasoning: Pitfalls and Paths to Reproducibilitycited by 90
  6. 2025Nemotron 3 Nano: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoningcited by 89
  7. 2025NaturalReasoning: Reasoning in the Wild with 2.8M Challenging Questionscited by 81
  8. 2025Dream-Coder 7B: An Open Diffusion Language Model for Codecited by 80
  9. 2025SPICE: Self-Play In Corpus Environments Improves Reasoningcited by 80
  10. 2025Practical Efficiency of Muon for Pretrainingcited by 78
  11. 2025Organize the Web: Constructing Domains Enhances Pre-Training Data Curationcited by 75
  12. 2025Don't be lazy: CompleteP enables compute-efficient deep transformerscited by 72
  13. 2024Blind Baselines Beat Membership Inference Attacks for Foundation Modelscited by 67
  14. 2025Nemotron-CLIMB: CLustering-based Iterative Data Mixture Bootstrapping for Language Model Pre-trainingcited by 65
  15. 2024Liquid: Language Models are Scalable and Unified Multi-modal Generatorscited by 63
  16. 2025Meta CLIP 2: A Worldwide Scaling Recipecited by 62
  17. 2025SuperBPE: Space Travel for Language Modelscited by 61
  18. 2025End-to-End Test-Time Training for Long Contextcited by 57
  19. 2025Optimizing Large Language Model Training Using FP4 Quantizationcited by 55
  20. 2024Visual Haystacks: A Vision-Centric Needle-In-A-Haystack Benchmarkcited by 53
  21. 2025MiniCPM4: Ultra-Efficient LLMs on End Devicescited by 51
  22. 2025Scaling Laws for Optimal Data Mixturescited by 50
  23. 2024Collapse or Thrive? Perils and Promises of Synthetic Data in a Self-Generating Worldcited by 49
  24. 2024Scaling Laws for Predicting Downstream Performance in LLMscited by 48
  25. 2025MegaMath: Pushing the Limits of Open Math Corporacited by 48
  26. 2025Scaling Laws for Native Multimodal Modelscited by 48
  27. 2024Improving Pretraining Data Using Perplexity Correlationscited by 46
  28. 2025Rethinking Reflection in Pre-Trainingcited by 43
  29. 2025mmBERT: A Modern Multilingual Encoder with Annealed Language Learningcited by 42
  30. 2024Data Selection via Optimal Control for Language Modelscited by 38
  31. 2024Programming Every Example: Lifting Pre-training Data Quality Like Experts at Scalecited by 37
  32. 2024InfiMM-WebMath-40B: Advancing Multimodal Pre-Training for Enhanced Mathematical Reasoningcited by 36
  33. 2025Ultra-FineWeb: Efficient Data Filtering and Verification for High-Quality LLM Training Datacited by 36
  34. 2025Seq vs Seq: An Open Suite of Paired Encoders and Decoderscited by 36
  35. 2024Training on the Test Task Confounds Evaluation and Emergencecited by 34
  36. 2024Surveying the Effects of Quality, Diversity, and Complexity in Synthetic Data From Large Language Modelscited by 33
  37. 2024Establishing Task Scaling Laws via Compute-Efficient Model Ladderscited by 33
  38. 2025Enhancing Multilingual LLM Pretraining with Model-Based Data Selectioncited by 33

50 most cited of 390 All 390 (JSON)