mlsanityData to 14 Oct 2026

Essential-Web v1.0: 24T tokens of organized web data

arXiv 2506.14111 ↗First posted 17 Jun 2025cs.CL cs.AI cs.LGcited by 8

Not on the map. Why?

Abstract (arXiv)

Data plays the most prominent role in how language models acquire skills and knowledge. The lack of massive, well-organized pre-training datasets results in costly and inaccessible data pipelines. We present Essential-Web v1.0, a 24-trillion-token dataset in which every document is annotated with a twelve-category taxonomy covering topic, format, content complexity, and quality. Taxonomy labels are produced by EAI-Distill-0.5b, a fine-tuned 0.5b-parameter model that achieves an annotator agreement within 3% of Qwen2.5-32B-Instruct. With nothing more than SQL-style filters, we obtain competitive web-curated datasets in math (-8.0% relative to SOTA), web code (+14.3%), STEM (+24.5%) and medical (+8.6%). Essential-Web v1.0 is available on HuggingFace: https://huggingface.co/datasets/EssentialAI/essential-web-v1.0

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Builds on 26 of its 43 references that are in the map. First posted on arXiv 17 Jun 2025.

Builds on

26 of 43 references.

References · 26

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  5. 2020Scaling Laws for Neural Language Modelscited by 6,586
  6. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  7. 2021Program Synthesis with Large Language Modelscited by 3,637
  8. 2016Enriching Word Vectors with Subword Informationcited by 1,583
  9. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,391
  10. 2020What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Examscited by 1,356
  11. 2019PubMedQA: A Dataset for Biomedical Research Question Answeringcited by 1,288
  12. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,135
14 more
  1. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 763
  2. 2022MedMCQA : A Large-scale Multi-Subject Multi-Choice Dataset for Medical domain Question Answeringcited by 543
  3. 2022Beyond neural scaling laws: beating power law scaling via data pruningcited by 528
  4. 2024Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Researchcited by 483
  5. 2021Explaining Neural Scaling Lawscited by 363
  6. 2024DeepSeek-Coder-V2: Breaking the Barrier of Closed-Source Models in Code Intelligencecited by 331
  7. 2024RedPajama: an Open Dataset for Training Large Language Modelscited by 182
  8. 2024Nemotron-CC: Transforming Common Crawl into a Refined Long-Horizon Pretraining Datasetcited by 160
  9. 2022Learning by Distilling Contextcited by 138
  10. 2025NaturalReasoning: Reasoning in the Wild with 2.8M Challenging Questionscited by 84
  11. 2025Organize the Web: Constructing Domains Enhances Pre-Training Data Curationcited by 76
  12. 2025MegaMath: Pushing the Limits of Open Math Corporacited by 51
  13. 2025Automatic Evaluation of Healthcare LLMs Beyond Question-Answeringcited by 20
  14. 2024Scaling Laws for Linear Complexity Language Modelscited by 19

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 6.

8 papers.

  1. 2025Language Models Improve When Pretraining Data Matches Target Taskscited by 19
  2. 2026Shaping capabilities with token-level data filteringcited by 13
  3. 2026Scaling Laws for Mixture Pretraining Under Data Constraintscited by 6
  4. 2025Thinking Augmented Pre-trainingcited by 4
  5. 2026Domain-Aware Scaling Laws Uncover Data Synergycited by 2
  6. 2026Optimal Splitting of Language Models from Mixtures to Specialized Domainscited by 1
  7. 2026daVinci-LLM:Towards the Science of Pretrainingcited by 0
  8. 2026T-Search: An Open Agentic Retriever and Playground for Hard Multi-Step Searchcited by 0