# When Does Generating More Help? Disentangling Fixed-Source Synthesis from Source Expansion in Synthetic Data Scaling

arXiv: https://arxiv.org/abs/2607.01727 · HTML: https://mlsanity.com/paper/2607.01727/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 18 of its 26 references that are in the map.
- First posted on arXiv 2 Jul 2026.

## Builds on (most cited first, 18 of 18)

- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Self-Instruct: Aligning Language Models with Self-Generated Instructions](https://mlsanity.com/paper/2212.10560/) (2022)
- [WizardLM: Empowering large pre-trained language models to follow complex instructions](https://mlsanity.com/paper/2304.12244/) (2023)
- [Deep Learning Scaling is Predictable, Empirically](https://mlsanity.com/paper/1712.00409/) (2017)
- [Scaling Synthetic Data Creation with 1,000,000,000 Personas](https://mlsanity.com/paper/2406.20094/) (2024)
- [What Makes Good Data for Alignment? A Comprehensive Study of Automatic Data Selection in Instruction Tuning](https://mlsanity.com/paper/2312.15685/) (2023)
- [Scaling Laws for Transfer](https://mlsanity.com/paper/2102.01293/) (2021)
- [OpenMathInstruct-2: Accelerating AI for Math with Massive Open-Source Instruction Data](https://mlsanity.com/paper/2410.01560/) (2024)
- [MAmmoTH2: Scaling Instructions from the Web](https://mlsanity.com/paper/2405.03548/) (2024)
- [The Quantization Model of Neural Scaling](https://mlsanity.com/paper/2303.13506/) (2023)
- [Distillation Scaling Laws](https://mlsanity.com/paper/2502.08606/) (2025)
- [Smaller, Weaker, Yet Better: Training LLM Reasoners via Compute-Optimal Sampling](https://mlsanity.com/paper/2408.16737/) (2024)
- [Scaling Laws of Synthetic Data for Language Models](https://mlsanity.com/paper/2503.19551/) (2025)
- [SuperCorrect: Advancing Small LLM Reasoning with Thought Template Distillation and Self-Correction](https://mlsanity.com/paper/2410.09008/) (2024)
- [Selecting Large Language Model to Fine-tune via Rectified Scaling Law](https://mlsanity.com/paper/2402.02314/) (2024)
- [TAGCOS: Task-agnostic Gradient Clustered Coreset Selection for Instruction Tuning Data](https://mlsanity.com/paper/2407.15235/) (2024)
- [Prescriptive Scaling Laws for Data Constrained Training](https://mlsanity.com/paper/2605.01640/) (2026)
- [Harnessing Negative Signals: Reinforcement Distillation from Teacher Data for LLM Reasoning](https://mlsanity.com/paper/2505.24850/) (2025)

## Abstract (arXiv)

Synthetic data can be scaled along two routes: Source Expansion (SE), which enlarges the source by adding seed materials or generators, and Fixed-Source Synthesis (FSS), which holds the source fixed and scales the generation budget. Existing scaling studies typically expand the source as the data grows, conflating SE with FSS and leaving FSS underexplored. We isolate FSS by holding the seed-question pool and teacher model fixed, varying only the per-question response budget under Rejection Sampling (RS). We adapt the rectified scaling law to FSS, deriving it from how repeated sampling covers a fixed source. Empirically, the derived form, fit on low budgets, predicts performance at the held-out highest budget for every evaluated teacher--student pair. At matched total-sample budgets, SE and FSS are comparable at small budgets; at large budgets, adding seed questions outperforms spending the same budget on more responses. Within FSS, however, neither synthesizing additional questions from the existing seeds nor varying the synthesis protocol outperforms plain RS at matched budgets. FSS is thus a bounded scaling axis and a controlled setting for comparing synthesis protocols. We will release our code and data to facilitate further research.