# The Common Pile v0.1: An 8TB Dataset of Public Domain and Openly Licensed Text

arXiv: https://arxiv.org/abs/2506.05209 · HTML: https://mlsanity.com/paper/2506.05209/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Jun 2025.

## Built on by (most cited first, 30 of 31)

- [Selective KV-Cache Sharing to Mitigate Timing Side-Channels in LLM Inference](https://mlsanity.com/paper/2508.08438/) (2025)
- [Shaping capabilities with token-level data filtering](https://mlsanity.com/paper/2601.21571/) (2026)
- [Does higher interpretability imply better utility? A Pairwise Analysis on Sparse Autoencoders](https://mlsanity.com/paper/2510.03659/) (2025)
- [DLM-Scope: Mechanistic Interpretability of Diffusion Language Models via Sparse Autoencoders](https://mlsanity.com/paper/2602.05859/) (2026)
- [TokSuite: Measuring the Impact of Tokenizer Choice on Language Model Behavior](https://mlsanity.com/paper/2512.20757/) (2025)
- [The German Commons - 154 Billion Tokens of Openly Licensed Text for German Language Models](https://mlsanity.com/paper/2510.13996/) (2025)
- [Data-Centric Lessons To Improve Speech-Language Pretraining](https://mlsanity.com/paper/2510.20860/) (2025)
- [K2-V2: A 360-Open, Reasoning-Enhanced LLM](https://mlsanity.com/paper/2512.06201/) (2025)
- [Rethinking Data Curation in LLM Training: Online Reweighting Offers Better Generalization than Offline Methods](https://mlsanity.com/paper/2605.05227/) (2026)
- [Contemporary AI lacks the imagination to diverge or negate in science](https://mlsanity.com/paper/2606.08251/) (2026)
- [TildeOpen LLM: Leveraging Curriculum Learning to Achieve Equitable Language Representation](https://mlsanity.com/paper/2603.08182/) (2026)
- [Security--Fidelity Tradeoffs: The Hidden Cost of Prompt Injection Defense](https://mlsanity.com/paper/2606.30783/) (2026)
- [SpreadsheetArena: Decomposing Preference in LLM Generation of Spreadsheet Workbooks](https://mlsanity.com/paper/2603.10002/) (2026)
- [LLMs Can Leak Training Data But Do They Want To? A Propensity-Aware Evaluation of Memorization in LLMs](https://mlsanity.com/paper/2606.06286/) (2026)
- [emb-diversity: A Tool for Embedding-Based Measurement of Data Diversity](https://mlsanity.com/paper/2607.19848/) (2026)
- [The Announcement Carries the Cue: Markup, Boundaries, and the Notation of Pre-Training Corpora](https://mlsanity.com/paper/2608.09093/) (2026)
- [The Emerging AI Paper-Review Arms Race: Adversarial Co-Evolution in Scholarly Publishing](https://mlsanity.com/paper/2609.07713/) (2026)
- [Perturb Your Data: Paraphrase-Guided Training Data Watermarking](https://mlsanity.com/paper/2512.17075/) (2025)
- [Selecting Language Models for Social Science: Start Small, Start Open, and Validate](https://mlsanity.com/paper/2601.10926/) (2026)
- [WIST: Web-Grounded Iterative Self-Play Tree for Domain-Targeted Reasoning Improvement](https://mlsanity.com/paper/2603.22352/) (2026)
- [Detecting Non-Membership in LLM Training Data via Rank Correlations](https://mlsanity.com/paper/2603.22707/) (2026)
- [Discovering Novel LLM Experts via Task-Capability Coevolution](https://mlsanity.com/paper/2604.14969/) (2026)
- [A Text-Steerable Instrument for Sketching Procedural Soundscapes via Language Models](https://mlsanity.com/paper/2607.00309/) (2026)
- [Epiplexity Guided Data Selection and Generation for Out-of-Distribution Generalization](https://mlsanity.com/paper/2608.11746/) (2026)
- [Directing large language models to follow the letter or spirit of the law](https://mlsanity.com/paper/2609.23083/) (2026)
- [It's All Training: A Fully Synthetic Single-Stage Recipe for LLMs](https://mlsanity.com/paper/2609.37891/) (2026)
- [Mitigating Private Data Leakage in LLMs with Whiteout](https://mlsanity.com/paper/2610.02418/) (2026)
- [Length Generalization Needs Proper Regularization](https://mlsanity.com/paper/2610.04518/) (2026)
- [Differentially Private Mixing of Public Datasets Improves Private Learning](https://mlsanity.com/paper/2610.06636/) (2026)
- [Project Greenhouse: Progress Toward Fully Open and Sovereign Agentic Search](https://mlsanity.com/paper/2610.11922/) (2026)

## Abstract (arXiv)

Large language models (LLMs) are typically trained on enormous quantities of unlicensed text, a practice that has led to scrutiny due to possible intellectual property infringement and ethical concerns. Training LLMs on openly licensed text presents a first step towards addressing these issues, but prior data collection efforts have yielded datasets too small or low-quality to produce performant LLMs. To address this gap, we collect, curate, and release the Common Pile v0.1, an eight terabyte collection of openly licensed text designed for LLM pretraining. The Common Pile comprises content from 30 sources that span diverse domains including research papers, code, books, encyclopedias, educational materials, audio transcripts, and more. Crucially, we validate our efforts by training two 7 billion parameter LLMs on text from the Common Pile: Comma v0.1-1T and Comma v0.1-2T, trained on 1 and 2 trillion tokens respectively. Both models attain competitive performance to LLMs trained on unlicensed text with similar computational budgets, such as Llama 1 and 2 7B. In addition to releasing the Common Pile v0.1 itself, we also release the code used in its creation as well as the training mixture and checkpoints for the Comma v0.1 models.