mlsanityData to 7 Oct 2026

Nemotron-CC: Transforming Common Crawl into a Refined Long-Horizon Pretraining Dataset

arXiv 2412.02595 ↗First posted 3 Dec 2024cs.CLcited by 153

Not on the map. Why?

Cited by 21 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Recent English Common Crawl datasets like FineWeb-Edu and DCLM achieved significant benchmark gains via aggressive model-based filtering, but at the cost of removing 90% of data. This limits their suitability for long token horizon training, such as 15T tokens for Llama 3.1. In this paper, we show how to achieve better trade-offs between accuracy and data quantity by a combination of classifier ensembling, synthetic data rephrasing, and reduced reliance on heuristic filters. When training 8B parameter models for 1T tokens, using a high-quality subset of our data improves MMLU by 5.6 over DCLM, demonstrating the efficacy of our methods for boosting accuracies over a relatively short token horizon. Furthermore, our full 6.3T token dataset matches DCLM on MMLU, but contains four times more unique real tokens than DCLM. This unlocks state-of-the-art training over a long token horizon: an 8B parameter model trained for 15T tokens, of which 7.2T came from our dataset, is better than the Llama 3.1 8B model: +5 on MMLU, +3.1 on ARC-Challenge, and +0.5 on average across ten diverse tasks. The dataset is available at https://data.commoncrawl.org/contrib/Nemotron/Nemotron-CC/index.html

Key facts. Cited by 153 papers in the map (data through 7 Oct 2026). Builds on 35 of its 54 references that are in the map. First posted on arXiv 3 Dec 2024.

Builds on

35 of 54 references.

References · 35

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  4. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  5. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  6. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  7. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  8. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  9. 2024Qwen2 Technical Reportcited by 2,076
  10. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,928
  11. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
  12. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
23 more
  1. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
  2. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
  3. 2020GLU Variants Improve Transformercited by 1,724
  4. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  5. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  6. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,104
  7. 2016Bag of Tricks for Efficient Text Classificationcited by 869
  8. 2017RACE: Large-scale ReAding Comprehension Dataset From Examinationscited by 851
  9. 2021Deduplicating Training Data Makes Language Models Bettercited by 705
  10. 2019ELI5: Long Form Question Answeringcited by 567
  11. 2023Textbooks Are All You Need II: phi-1.5 technical reportcited by 559
  12. 2019CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Datacited by 544
  13. 2024Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Researchcited by 475
  14. 2023TinyStories: How Small Can Language Models Be and Still Speak Coherent English?cited by 425
  15. 2024DataComp-LM: In search of the next generation of training sets for language modelscited by 381
  16. 2016FastText.zip: Compressing text classification modelscited by 331
  17. 2023MADLAD-400: A Multilingual And Document-Level Large Audited Datasetcited by 180
  18. 2024Rephrasing the Web: A Recipe for Compute and Data-Efficient Language Modelingcited by 156
  19. 2022Towards a Cleaner Document-Oriented Multilingual Crawled Corpuscited by 143
  20. 2024Instruction Pre-Training: Language Models are Supervised Multitask Learnerscited by 63
  21. 2024Arctic-Embed: Scalable, Efficient, and Accurate Text Embedding Modelscited by 60
  22. 2024Zyda-2: a 5 Trillion Token High-Quality Datasetcited by 12
  23. 2024Data, Data Everywhere: A Guide for Pretraining Dataset Constructioncited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 80; 2026: 71.

153 papers.

Sort:

By area:

  1. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMscited by 1,193
  2. 2025GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation Modelscited by 446
  3. 2025Kimi-VL Technical Reportcited by 381
  4. 2025OpenThoughts: Data Recipes for Reasoning Modelscited by 287
  5. 2025Scaling Latent Reasoning via Looped Language Modelscited by 145
  6. 2025Emu3.5: Native Multimodal Models are World Learnerscited by 119
  7. 2025NVIDIA Nemotron Nano 2: An Accurate and Efficient Hybrid Mamba-Transformer Reasoning Modelcited by 94
  8. 2025Nemotron-H: A Family of Accurate and Efficient Hybrid Mamba-Transformer Modelscited by 88
  9. 2025Nemotron 3 Nano: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoningcited by 85
  10. 2025Self-Adapting Language Modelscited by 71
  11. 2025Nemotron-CLIMB: CLustering-based Iterative Data Mixture Bootstrapping for Language Model Pre-trainingcited by 62
  12. 2025Intern-S1: A Scientific Multimodal Foundation Modelcited by 49
38 more
  1. 2025MegaMath: Pushing the Limits of Open Math Corporacited by 47
  2. 2025Diffusion Language Models are Super Data Learnerscited by 45
  3. 2025Jet-Nemotron: Efficient Language Model with Post Neural Architecture Searchcited by 39
  4. 2025Nemotron-CC-Math: A 133 Billion-Token-Scale High Quality Math Pretraining Datasetcited by 34
  5. 2025Scaling Behavior of Discrete Diffusion Language Modelscited by 32
  6. 2025Recycling the Web: A Method to Enhance Pre-training Data Quality and Quantity for Language Modelscited by 31
  7. 2024Maximize Your Data's Potential: Enhancing LLM Accuracy with Two-Phase Pretrainingcited by 24
  8. 2025Pre-training under infinite computecited by 24
  9. 2025Primus: A Pioneering Collection of Open-Source Datasets for Cybersecurity LLM Trainingcited by 23
  10. 2025Training Optimal Large Diffusion Language Modelscited by 21
  11. 2025Kanana: Compute-efficient Bilingual Language Modelscited by 19
  12. 2025BeyondWeb: Lessons from Scaling Synthetic Data for Trillion-scale Pretrainingcited by 19
  13. 2025Language Models Improve When Pretraining Data Matches Target Taskscited by 18
  14. 2025Mid-Training of Large Language Models: A Surveycited by 18
  15. 2025Rewriting Pre-Training Data Boosts LLM Performance in Math and Codecited by 17
  16. 2025SWAN-GPT: An Efficient and Scalable Approach for Long-Context Language Modelingcited by 16
  17. 2025MuLoCo: Muon is a practical inner optimizer for DiLoCocited by 16
  18. 2025Data Mixing Can Induce Phase Transitions in Knowledge Acquisitioncited by 15
  19. 2025RefineX: Learning to Refine Pre-training Data at Scale from Expert-Guided Programscited by 15
  20. 2025Dynamic Large Concept Models: Latent Reasoning in an Adaptive Semantic Spacecited by 14
  21. 2025Training Dynamics Impact Post-Training Quantization Robustnesscited by 13
  22. 2026NextFlow: Unified Sequential Modeling Activates Multimodal Understanding and Generationcited by 13
  23. 2025AgentFrontier: Expanding the Capability Frontier of LLM Agents with ZPD-Guided Data Synthesiscited by 12
  24. 2025How Learning Rate Decay Wastes Your Best Data in Curriculum-Based LLM Pretrainingcited by 12
  25. 2025ModernBERT or DeBERTaV3? Examining Architecture and Data Influence on Transformer Encoder Models Performancecited by 11
  26. 2025Reformulation for Pretraining Data Augmentationcited by 9
  27. 2025textttDroid: A Resource Suite for AI-Generated Code Detectioncited by 9
  28. 2025Mixture of Neuron Expertscited by 7
  29. 2025Detecting Data Contamination in LLMs via In-Context Learningcited by 7
  30. 2026EuroLLM-22B: Technical Reportcited by 7
  31. 2026Data-efficient pre-training by scaling synthetic megadocscited by 7
  32. 2025GneissWeb: Preparing High Quality Data for LLMs at Scalecited by 6
  33. 2025Register Always Matters: Analysis of LLM Pretraining Data Through the Lens of Language Variationcited by 6
  34. 2025Aleph-Alpha-GermanWeb: Improving German-language LLM pre-training with model-based data curation and synthetic data generationcited by 6
  35. 2025Essential-Web v1.0: 24T tokens of organized web datacited by 6
  36. 2025Open-sci-ref-0.01: open and reproducible reference baselines for language model and dataset comparisoncited by 6
  37. 2025Training Matryoshka Mixture-of-Experts for Elastic Inference-Time Expert Utilizationcited by 6
  38. 2025RePro: Training Language Models to Faithfully Recycle the Web for Pretrainingcited by 6

50 most cited of 153 All 153 (JSON)