# BeyondWeb: Lessons from Scaling Synthetic Data for Trillion-scale Pretraining

arXiv: https://arxiv.org/abs/2508.10975 · HTML: https://mlsanity.com/paper/2508.10975/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Placed in LLM synthetic data generation, one of 13 directions in Synthetic data and self-improvement for LLMs.
- Builds on 36 of its 46 references that are in the map.
- First posted on arXiv 14 Aug 2025.

## Placed in

- Synthetic data and self-improvement for LLMs / [LLM synthetic data generation](https://mlsanity.com/synth/llm-synthetic-data-generation/)

## Builds on (most cited first, 30 of 36)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [HellaSwag: Can a Machine Really Finish Your Sentence?](https://mlsanity.com/paper/1905.07830/) (2019)
- [PIQA: Reasoning about Physical Commonsense in Natural Language](https://mlsanity.com/paper/1911.11641/) (2019)
- [BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions](https://mlsanity.com/paper/1905.10044/) (2019)
- [Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering](https://mlsanity.com/paper/1809.02789/) (2018)
- [CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge](https://mlsanity.com/paper/1811.00937/) (2018)
- [Qwen2.5-Coder Technical Report](https://mlsanity.com/paper/2409.12186/) (2024)
- [The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale](https://mlsanity.com/paper/2406.17557/) (2024)
- [RACE: Large-scale ReAding Comprehension Dataset From Examinations](https://mlsanity.com/paper/1704.04683/) (2017)
- [Crowdsourcing Multiple Choice Science Questions](https://mlsanity.com/paper/1707.06209/) (2017)
- [Phi-4 Technical Report](https://mlsanity.com/paper/2412.08905/) (2024)
- [Textbooks Are All You Need II: phi-1.5 technical report](https://mlsanity.com/paper/2309.05463/) (2023)
- [TinyStories: How Small Can Language Models Be and Still Speak Coherent English?](https://mlsanity.com/paper/2305.07759/) (2023)
- [Scaling Data-Constrained Language Models](https://mlsanity.com/paper/2305.16264/) (2023)
- [DataComp-LM: In search of the next generation of training sets for language models](https://mlsanity.com/paper/2406.11794/) (2024)
- [The Curse of Recursion: Training on Generated Data Makes Models Forget](https://mlsanity.com/paper/2305.17493/) (2023)
- [2 OLMo 2 Furious](https://mlsanity.com/paper/2501.00656/) (2024)
- [Physics of Language Models: Part 3.1, Knowledge Storage and Extraction](https://mlsanity.com/paper/2309.14316/) (2023)
- [Pretraining Language Models with Human Preferences](https://mlsanity.com/paper/2302.08582/) (2023)
- [RedPajama: an Open Dataset for Training Large Language Models](https://mlsanity.com/paper/2411.12372/) (2024)
- [Rephrasing the Web: A Recipe for Compute and Data-Efficient Language Modeling](https://mlsanity.com/paper/2401.16380/) (2024)
- [Nemotron-CC: Transforming Common Crawl into a Refined Long-Horizon Pretraining Dataset](https://mlsanity.com/paper/2412.02595/) (2024)
- [Self-Alignment with Instruction Backtranslation](https://mlsanity.com/paper/2308.06259/) (2023)
- [OLMES: A Standard for Language Model Evaluations](https://mlsanity.com/paper/2406.08446/) (2024)
- [The Curious Decline of Linguistic Diversity: Training Language Models on Synthetic Text](https://mlsanity.com/paper/2311.09807/) (2023)
- [Large Language Models Suffer From Their Own Output: An Analysis of the Self-Consuming Training Loop](https://mlsanity.com/paper/2311.16822/) (2023)
- [Scaling Laws for Data Filtering -- Data Curation cannot be Compute Agnostic](https://mlsanity.com/paper/2404.07177/) (2024)
- [Organize the Web: Constructing Domains Enhances Pre-Training Data Curation](https://mlsanity.com/paper/2502.10341/) (2025)

## Built on by (most cited first, 21 of 21)

- [Arcee Trinity Large Technical Report](https://mlsanity.com/paper/2602.17004/) (2026)
- [Why Larger Models Learn More: Effects of Capacity, Interference, and Rare-Task Retention](https://mlsanity.com/paper/2605.29548/) (2026)
- [RePro: Training Language Models to Faithfully Recycle the Web for Pretraining](https://mlsanity.com/paper/2510.10681/) (2025)
- [Towards Active Synthetic Data Generation for Finetuning Language Models](https://mlsanity.com/paper/2512.00884/) (2025)
- [propella-1: Multi-Property Document Annotation for LLM Data Curation at Scale](https://mlsanity.com/paper/2602.12414/) (2026)
- [How Can We Synthesize High-Quality Pretraining Data? A Systematic Study of Prompt Design, Generator Model, and Source Data](https://mlsanity.com/paper/2604.13977/) (2026)
- [Self-Improvement of Large Language Models: A Technical Overview and Future Outlook](https://mlsanity.com/paper/2603.25681/) (2026)
- [Linguistics and Human Brain: A Perspective of Computational Neuroscience](https://mlsanity.com/paper/2602.08275/) (2026)
- [Data-Centric Lessons To Improve Speech-Language Pretraining](https://mlsanity.com/paper/2510.20860/) (2025)
- [Understanding Data Temporality Impact on Large Language Models Pre-training](https://mlsanity.com/paper/2605.22769/) (2026)
- [20/20 Vision Language Models: A Prescription for Better VLMs through Data Curation Alone](https://mlsanity.com/paper/2605.11405/) (2026)
- [Generating Pretraining Tokens from Organic Data for Data-Bound Scaling](https://mlsanity.com/paper/2605.17849/) (2026)
- [Bridging Compute- and Data-Optimal Pretraining](https://mlsanity.com/paper/2607.25271/) (2026)
- [Thomson: Continual Learning of Frontier Models for SovereignAI](https://mlsanity.com/paper/2608.27147/) (2026)
- [Synthetic Mixed Training: Scaling Parametric Knowledge Acquisition Beyond RAG](https://mlsanity.com/paper/2603.23562/) (2026)
- [DataOrchestra: Learning to Orchestrate Per-Example Curation of Pretraining Data](https://mlsanity.com/paper/2607.24717/) (2026)
- [The Announcement Carries the Cue: Markup, Boundaries, and the Notation of Pre-Training Corpora](https://mlsanity.com/paper/2608.09093/) (2026)
- [BYOL: Bring Your Own Language Into LLMs](https://mlsanity.com/paper/2601.10804/) (2026)
- [Continually self-improving AI](https://mlsanity.com/paper/2603.18073/) (2026)
- [WRAP++: Web discoveRy Amplified Pretraining](https://mlsanity.com/paper/2604.06829/) (2026)
- [It's All Training: A Fully Synthetic Single-Stage Recipe for LLMs](https://mlsanity.com/paper/2609.37891/) (2026)

## Abstract (arXiv)

Recent advances in large language model (LLM) pretraining have shown that simply scaling data quantity eventually leads to diminishing returns, hitting a data wall. In response, the use of synthetic data for pretraining has emerged as a promising paradigm for pushing the frontier of performance. Despite this, the factors affecting synthetic data quality remain poorly understood. In this work, we introduce BeyondWeb, a synthetic data generation framework that produces high-quality synthetic data for pretraining. BeyondWeb significantly extends the capabilities of traditional web-scale datasets, outperforming state-of-the-art synthetic pretraining datasets such as Cosmopedia and Nemotron-CC's high-quality synthetic subset (Nemotron-Synth) by up to 5.1 percentage points (pp) and 2.6pp, respectively, when averaged across a suite of 14 benchmark evaluations. It delivers up to 7.7x faster training than open web data and 2.7x faster than Nemotron-Synth. Remarkably, a 3B model trained for 180B tokens on BeyondWeb outperforms an 8B model trained for the same token budget on Cosmopedia. We also present several insights from BeyondWeb on synthetic data for pretraining: what drives its benefits, which data to rephrase and how, and the impact of model size and family on data quality. Overall, our work shows that there's no silver bullet for generating high-quality synthetic pretraining data. The best outcomes require jointly optimizing many factors, a challenging task that requires rigorous science and practical expertise. Naive approaches can yield modest improvements, potentially at great cost, while well-executed methods can yield transformative improvements, as exemplified by BeyondWeb.