mlsanityData to 14 Oct 2026

Zyda-2: a 5 Trillion Token High-Quality Dataset

arXiv 2411.06068 ↗First posted 9 Nov 2024cs.CL cs.AIcited by 13

Not on the map. Why?

Abstract (arXiv)

In this technical report, we present Zyda-2: a five trillion token dataset for language model pretraining. Zyda-2 was used to train our Zamba2 series of models which are state-of-the-art for their weight class. We build Zyda-2 by collating high-quality open-source tokens such as FineWeb and DCLM, then distilling them to the highest-quality subset via cross-deduplication and model-based quality filtering. Zyda-2 is released under a permissive open license, and is available at https://huggingface.co/datasets/Zyphra/Zyda-2

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Builds on 18 of its 19 references that are in the map. First posted on arXiv 9 Nov 2024.

Builds on

18 of 19 references.

References · 18

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  3. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  4. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,917
  5. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,160
  6. 2024Gemma: Open Models Based on Gemini Research and Technologycited by 916
  7. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 763
  8. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 723
  9. 2021Deduplicating Training Data Makes Language Models Bettercited by 714
  10. 2024Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Researchcited by 483
  11. 2024DataComp-LM: In search of the next generation of training sets for language modelscited by 398
  12. 2024Rephrasing the Web: A Recipe for Compute and Data-Efficient Language Modelingcited by 161
6 more
  1. 2024Zamba: A Compact 7B SSM Hybrid Modelcited by 124
  2. 2024Perplexed by Perplexity: Perplexity-Based Data Pruning With Small Reference Modelscited by 84
  3. 2024Does your data spark joy? Performance gains from domain upsampling at the end of trainingcited by 52
  4. 2024BlackMamba: Mixture of Experts for State-Space Modelscited by 37
  5. 2024Nemotron-4 15B Technical Reportcited by 36
  6. 2024Zyda: A 1.3T Dataset for Open Language Modelingcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 8; 2026: 4.

13 papers.

Sort:

By area:

  1. 2024Nemotron-CC: Transforming Common Crawl into a Refined Long-Horizon Pretraining Datasetcited by 160
  2. 2025MegaMath: Pushing the Limits of Open Math Corporacited by 51
  3. 2025Nemotron-CC-Math: A 133 Billion-Token-Scale High Quality Math Pretraining Datasetcited by 38
  4. 2025Language Models Improve When Pretraining Data Matches Target Taskscited by 19
  5. 2025Nemotron-Flash: Towards Latency-Optimal Hybrid Small Language Modelscited by 15
  6. 2025MoNE: Replacing Redundant Experts with Lightweight Novices for Structured Pruning of MoEcited by 13
  7. 2025Datasets, Documents, and Repetitions: The Practicalities of Unequal Data Qualitycited by 8
  8. 2025GneissWeb: Preparing High Quality Data for LLMs at Scalecited by 7
  9. 2025A Partition Cover Approach to Tokenizationcited by 5
  10. 2026Tokenizer-Agnostic Engram Modulecited by 1
  11. 2026Scaling Inherently Interpretable Language Modelscited by 1
  12. 2026Scaling Adaptive Depth with Norm-Agnostic Residual Networkscited by 0
1 more
  1. 2026FlexMoE: One-for-All Nested Intra-Expert Pruning for MoE Language Modelscited by 0