# Zyda-2: a 5 Trillion Token High-Quality Dataset

arXiv: https://arxiv.org/abs/2411.06068 · HTML: https://mlsanity.com/paper/2411.06068/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 19 references that are in the map.
- First posted on arXiv 9 Nov 2024.

## Builds on (most cited first, 18 of 18)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [The Pile: An 800GB Dataset of Diverse Text for Language Modeling](https://mlsanity.com/paper/2101.00027/) (2020)
- [Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone](https://mlsanity.com/paper/2404.14219/) (2024)
- [The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale](https://mlsanity.com/paper/2406.17557/) (2024)
- [Gemma: Open Models Based on Gemini Research and Technology](https://mlsanity.com/paper/2403.08295/) (2024)
- [The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only](https://mlsanity.com/paper/2306.01116/) (2023)
- [MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies](https://mlsanity.com/paper/2404.06395/) (2024)
- [Deduplicating Training Data Makes Language Models Better](https://mlsanity.com/paper/2107.06499/) (2021)
- [Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Research](https://mlsanity.com/paper/2402.00159/) (2024)
- [DataComp-LM: In search of the next generation of training sets for language models](https://mlsanity.com/paper/2406.11794/) (2024)
- [Rephrasing the Web: A Recipe for Compute and Data-Efficient Language Modeling](https://mlsanity.com/paper/2401.16380/) (2024)
- [Zamba: A Compact 7B SSM Hybrid Model](https://mlsanity.com/paper/2405.16712/) (2024)
- [Perplexed by Perplexity: Perplexity-Based Data Pruning With Small Reference Models](https://mlsanity.com/paper/2405.20541/) (2024)
- [Does your data spark joy? Performance gains from domain upsampling at the end of training](https://mlsanity.com/paper/2406.03476/) (2024)
- [BlackMamba: Mixture of Experts for State-Space Models](https://mlsanity.com/paper/2402.01771/) (2024)
- [Nemotron-4 15B Technical Report](https://mlsanity.com/paper/2402.16819/) (2024)
- [Zyda: A 1.3T Dataset for Open Language Modeling](https://mlsanity.com/paper/2406.01981/) (2024)

## Built on by (most cited first, 13 of 13)

- [Nemotron-CC: Transforming Common Crawl into a Refined Long-Horizon Pretraining Dataset](https://mlsanity.com/paper/2412.02595/) (2024)
- [MegaMath: Pushing the Limits of Open Math Corpora](https://mlsanity.com/paper/2504.02807/) (2025)
- [Nemotron-CC-Math: A 133 Billion-Token-Scale High Quality Math Pretraining Dataset](https://mlsanity.com/paper/2508.15096/) (2025)
- [Language Models Improve When Pretraining Data Matches Target Tasks](https://mlsanity.com/paper/2507.12466/) (2025)
- [Nemotron-Flash: Towards Latency-Optimal Hybrid Small Language Models](https://mlsanity.com/paper/2511.18890/) (2025)
- [MoNE: Replacing Redundant Experts with Lightweight Novices for Structured Pruning of MoE](https://mlsanity.com/paper/2507.00390/) (2025)
- [Datasets, Documents, and Repetitions: The Practicalities of Unequal Data Quality](https://mlsanity.com/paper/2503.07879/) (2025)
- [GneissWeb: Preparing High Quality Data for LLMs at Scale](https://mlsanity.com/paper/2502.14907/) (2025)
- [A Partition Cover Approach to Tokenization](https://mlsanity.com/paper/2501.06246/) (2025)
- [Tokenizer-Agnostic Engram Module](https://mlsanity.com/paper/2607.29065/) (2026)
- [Scaling Inherently Interpretable Language Models](https://mlsanity.com/paper/2608.07594/) (2026)
- [Scaling Adaptive Depth with Norm-Agnostic Residual Networks](https://mlsanity.com/paper/2606.16112/) (2026)
- [FlexMoE: One-for-All Nested Intra-Expert Pruning for MoE Language Models](https://mlsanity.com/paper/2606.27866/) (2026)

## Abstract (arXiv)

In this technical report, we present Zyda-2: a five trillion token dataset for language model pretraining. Zyda-2 was used to train our Zamba2 series of models which are state-of-the-art for their weight class. We build Zyda-2 by collating high-quality open-source tokens such as FineWeb and DCLM, then distilling them to the highest-quality subset via cross-deduplication and model-based quality filtering. Zyda-2 is released under a permissive open license, and is available at https://huggingface.co/datasets/Zyphra/Zyda-2