# A Deep Dive into Scaling RL for Code Generation with Synthetic Data and Curricula

arXiv: https://arxiv.org/abs/2603.24202 · HTML: https://mlsanity.com/paper/2603.24202/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Code and instruction data synthesis, one of 13 directions in Synthetic data and self-improvement for LLMs.
- Builds on 27 of its 31 references that are in the map.
- First posted on arXiv 25 Mar 2026.

## Placed in

- Synthetic data and self-improvement for LLMs / [Code and instruction data synthesis](https://mlsanity.com/synth/code-and-instruction-data-synthesis/)

## Builds on (most cited first, 27 of 27)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [DAPO: An Open-Source LLM Reinforcement Learning System at Scale](https://mlsanity.com/paper/2503.14476/) (2025)
- [LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code](https://mlsanity.com/paper/2403.07974/) (2024)
- [The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models](https://mlsanity.com/paper/2505.22617/) (2025)
- [Absolute Zero: Reinforced Self-play Reasoning with Zero Data](https://mlsanity.com/paper/2505.03335/) (2025)
- [Logic-RL: Unleashing LLM Reasoning with Rule-Based Reinforcement Learning](https://mlsanity.com/paper/2502.14768/) (2025)
- [Magicoder: Empowering Code Generation with OSS-Instruct](https://mlsanity.com/paper/2312.02120/) (2023)
- [Online Difficulty Filtering for Reasoning Oriented Reinforcement Learning](https://mlsanity.com/paper/2504.03380/) (2025)
- [CWM: An Open-Weights LLM for Research on Code Generation with World Models](https://mlsanity.com/paper/2510.02387/) (2025)
- [Intrinsically Motivated Goal Exploration Processes with Automatic Curriculum Learning](https://mlsanity.com/paper/1708.02190/) (2017)
- [GRPO-LEAD: A Difficulty-Aware Reinforcement Learning Approach for Concise Mathematical Reasoning in Language Models](https://mlsanity.com/paper/2504.09696/) (2025)
- [STP: Self-play LLM Theorem Provers with Iterative Conjecturing and Proving](https://mlsanity.com/paper/2502.00212/) (2025)
- [Training Large Language Models for Reasoning through Reverse Curriculum Reinforcement Learning](https://mlsanity.com/paper/2402.05808/) (2024)
- [Self-Challenging Language Model Agents](https://mlsanity.com/paper/2506.01716/) (2025)
- [DUMP: Automated Distribution-Level Curriculum Learning for RL-based LLM Post-training](https://mlsanity.com/paper/2504.09710/) (2025)
- [Learning Formal Mathematics From Intrinsic Motivation](https://mlsanity.com/paper/2407.00695/) (2024)
- [Synthetic Data Generation & Multi-Step RL for Reasoning & Tool Use](https://mlsanity.com/paper/2504.04736/) (2025)
- [SwS: Self-aware Weakness-driven Problem Synthesis in Reinforcement Learning for LLM Reasoning](https://mlsanity.com/paper/2506.08989/) (2025)
- [Curry-DPO: Enhancing Alignment using Curriculum Learning & Ranked Preferences](https://mlsanity.com/paper/2403.07230/) (2024)
- [Learning to Solve and Verify: A Self-Play Framework for Code and Test Generation](https://mlsanity.com/paper/2502.14948/) (2025)
- [Synthetic Data RL: Task Definition Is All You Need](https://mlsanity.com/paper/2505.17063/) (2025)
- [SPARQ: Synthetic Problem Generation for Reasoning via Quality-Diversity Algorithms](https://mlsanity.com/paper/2506.06499/) (2025)
- [Generative Data Refinement: Just Ask for Better Data](https://mlsanity.com/paper/2509.08653/) (2025)

## Built on by (most cited first, 3 of 3)

- [BenchEvolver: Frontier Task Synthesis via Solution-Centric Evolution](https://mlsanity.com/paper/2606.01286/) (2026)
- [AutoOR: Scalably Post-training LLMs to Autoformulate Operations Research Problems](https://mlsanity.com/paper/2604.16804/) (2026)
- [Training Large Language Models for Small-Molecule Design with Synthetic Task Scaling](https://mlsanity.com/paper/2609.04735/) (2026)

## Abstract (arXiv)

Reinforcement learning (RL) has emerged as a powerful paradigm for improving large language models beyond supervised fine-tuning, yet sustaining performance gains at scale remains an open challenge, as data diversity and structure, rather than volume alone, become the limiting factor. We address this by introducing a scalable multi-turn synthetic data generation pipeline in which a teacher model iteratively refines problems based on in-context student performance summaries, producing structured difficulty progressions without any teacher fine-tuning. Compared to single-turn generation, this multi-turn approach substantially improves the yield of valid synthetic problems and naturally produces stepping stones, i.e. easier and harder variants of the same core task, that support curriculum-based training. We systematically study how task difficulty, curriculum scheduling, and environment diversity interact during RL training across the Llama3.1-8B Instruct and Qwen3-8B Base model families, with additional scaling experiments on Qwen2.5-32B. Our results show that synthetic augmentation consistently improves in-domain code and in most cases out-of-domain math performance, and we provide empirical insights into how curriculum design and data diversity jointly shape RL training dynamics.