mlsanityData to 7 Oct 2026

Synthetic Data RL: Task Definition Is All You Need

arXiv 2505.17063 ↗First posted 18 May 2025cs.CL cs.AI cs.LGcited by 15

On the map: Synthetic data and self-improvement for LLMs → Synthetic reasoning data and bootstrapping

Abstract (arXiv)

Reinforcement learning (RL) is a powerful way to adapt foundation models to specialized tasks, but its reliance on large-scale human-labeled data limits broad adoption. We introduce Synthetic Data RL, a simple and general framework that reinforcement fine-tunes models using only synthetic data generated from a task definition. Our method first generates question and answer pairs from the task definition and retrieved documents, then adapts the difficulty of the question based on model solvability, and selects questions using the average pass rate of the model across samples for RL training. On Qwen-2.5-7B, our method achieves a 29.2% absolute improvement over the base model on GSM8K (+2.9 pp vs. instruction-tuned, +6.6 pp vs. Self-Instruct), 8.7% on MATH, 13.1% on GPQA (+7.0 pp vs. SynthLLM), 8.9% on MedQA, 17.7% on CQA (law) and 13.7% on CFA (finance). It surpasses supervised fine-tuning under the same data budget and nearly matches RL with full human data across datasets (e.g., +17.2 pp on GSM8K). Adding 100 human demonstrations improves the performance of GSM8K only by 0.4 pp, showing a limited added value. By reducing human data annotation, Synthetic Data RL enables scalable and efficient RL-based model adaptation. Code and demos are available at https://github.com/gydpku/Data_Synthesis_RL/.

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Placed in Synthetic reasoning data and bootstrapping, one of 13 directions in Synthetic data and self-improvement for LLMs. Builds on 33 of its 60 references that are in the map. First posted on arXiv 18 May 2025.

Builds on

33 of 60 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024RL on Incorrect Synthetic Data Scales the Efficiency of LLM Math Reasoning by Eight-Fold20 Jun
  2. 2023Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Models11 Dec

Other references · 31

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  4. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  5. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  6. 2020What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Examscited by 1,324
  7. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?cited by 948
  8. 2015Improving Neural Machine Translation Models with Monolingual Datacited by 784
  9. 2024Self-Rewarding Language Modelscited by 654
  10. 2023WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instructcited by 618
  11. 2025SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wildcited by 505
  12. 2025Cognitive Behaviors that Enable Self-Improving Reasoners, or, Four Habits of Highly Effective STaRscited by 373
19 more
  1. 2022CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learningcited by 342
  2. 2025Absolute Zero: Reinforced Self-play Reasoning with Zero Datacited by 323
  3. 2025Reinforcement Learning for Reasoning in Large Language Models with One Training Examplecited by 288
  4. 2024RAFT: Adapting Language Model to Domain Specific RAGcited by 258
  5. 2023PIXIU: A Large Language Model, Instruction Data and Evaluation Benchmark for Financecited by 217
  6. 2020Retrieve, Program, Repeat: Complex Knowledge Base Question Answering via Alternate Meta-learningcited by 216
  7. 2025A Minimalist Approach to LLM Reasoning: from Rejection Sampling to Reinforcecited by 141
  8. 2024Evaluation of OpenAI o1: Opportunities and Challenges of AGIcited by 127
  9. 2024DART-Math: Difficulty-Aware Rejection Tuning for Mathematical Problem-Solvingcited by 92
  10. 2024A Survey of Large Language Models in Finance (FinLLMs)cited by 77
  11. 2025Scaling Laws of Synthetic Data for Language Modelscited by 44
  12. 2023Prompt2Model: Generating Deployable Models from Natural Language Instructionscited by 40
  13. 2024Enhancing Multi-Step Reasoning Abilities of Language Models through Direct Q-Function Optimizationcited by 22
  14. 2025Climbing the Ladder of Reasoning: What LLMs Can-and Still Can't-Solve after SFT?cited by 21
  15. 2024SynthesizRR: Generating Diverse Datasets with Retrieval Augmentationcited by 13
  16. 2023PAXQA: Generating Cross-lingual Question Answering Examples at Training Scalecited by 10
  17. 2024Leveraging the Domain Adaptation of Retrieval Augmented Generation Models for Question Answering and Reducing Hallucinationcited by 7
  18. 2024Targeted synthetic data generation for tabular data via hardness characterizationcited by 4
  19. 2025NAACL2025 Tutorial: Adaptation of Large Language Modelscited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 8; 2026: 7.

15 papers.

Sort:

By area:

  1. 2025The Landscape of Agentic Reinforcement Learning for LLMs: A Surveycited by 195
  2. 2025Beyond Pass@1: Self-Play with Variational Problem Synthesis Sustains RLVRcited by 53
  3. 2025Search Self-play: Pushing the Frontier of Agent Capability without Supervisioncited by 31
  4. 2025CoT-Self-Instruct: Building high-quality synthetic prompts for reasoning and non-reasoning taskscited by 30
  5. 2025Routine: A Structural Planning Framework for LLM Agent System in Enterprisecited by 9
  6. 2025Learning to Pose Problems: Reasoning-Driven and Solver-Adaptive Data Synthesiscited by 8
  7. 2026SCALER:Synthetic Scalable Adaptive Learning Environment for Reasoningcited by 7
  8. 2026ReSyn: Autonomously Scaling Synthetic Environments for Reasoning Modelscited by 5
  9. 2026A Deep Dive into Scaling RL for Code Generation with Synthetic Data and Curriculacited by 3
  10. 2025Think Less, Label Better: Multi-Stage Domain-Grounded Synthetic Data Generation for Fine-Tuning Large Language Models in Telecommunicationscited by 1
  11. 2026MathMixup: Boosting LLM Mathematical Reasoning with Difficulty-Controllable Data Synthesis and Curriculum Learningcited by 1
  12. 2026Scaling Multi-Hop Training Data via Graph-Constrained Path Selectioncited by 1
3 more
  1. 2025LOCA: Logical Chain Augmentation for Scientific Corpus Cleaningcited by 0
  2. 2026From Atoms to Chains: Divergence-Guided Reasoning Curriculum for Unlabeled LLM Domain Adaptationcited by 0
  3. 2026Eliciting Medical Reasoning with Knowledge-enhanced Data Synthesis: A Semi-Supervised Reinforcement Learning Approachcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026S3-R1: Learning to Retrieve and Answer Step-by-Step with Synthetic Data
  2. 2025Webscale-RL: Automated Data Pipeline for Scaling RL Data to Pretraining Levels
  3. 2024TeaMs-RL: Teaching LLMs to Generate Better Instruction Datasets via Reinforcement Learning
  4. 2025Synthetic Data Generation Using Large Language Models: Advances in Text and Code
  5. 2026Synthetic Data for any Differentiable Target
  6. 2026Learning from Synthetic Data Improves Multi-hop Reasoning
  7. 2025Synthetic Data Generation & Multi-Step RL for Reasoning & Tool Use
  8. 2026RingSQL: Schema-Independent Synthetic Data Generation for Text-to-SQL Reinforcement Learning