mlsanityData to 7 Oct 2026

Synthetic Data Generation & Multi-Step RL for Reasoning & Tool Use

arXiv 2504.04736 ↗First posted 7 Apr 2025cs.AI cs.CL cs.LGcited by 45

On the map: LLM reasoning → Reinforcement learning for reasoning

also in 2 more
Abstract (arXiv)

Reinforcement learning has been shown to improve the performance of large language models. However, traditional approaches like RLHF or RLAIF treat the problem as single-step. As focus shifts toward more complex reasoning and agentic tasks, language models must take multiple steps of text generation, reasoning and environment interaction before generating a solution. We propose a synthetic data generation and RL methodology targeting multi-step optimization scenarios. This approach, called Step-Wise Reinforcement Learning (SWiRL), iteratively generates multi-step reasoning and tool use data, and then learns from that data. It employs a simple step-wise decomposition that breaks each multi-step trajectory into multiple sub-trajectories corresponding to each action by the original model. It then applies synthetic data filtering and RL optimization on these sub-trajectories. We evaluated SWiRL on a number of multi-step tool use, question answering, and mathematical reasoning tasks. Our experiments show that SWiRL outperforms baseline approaches by 21.5%, 12.3%, 14.8%, 11.1%, and 15.3% in relative accuracy on GSM8K, HotPotQA, CofCA, MuSiQue, and BeerQA, respectively. Excitingly, the approach exhibits generalization across tasks: for example, training only on HotPotQA (text question-answering) improves zero-shot performance on GSM8K (a math dataset) by a relative 16.9%.

Key facts. Cited by 45 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 41 of its 45 references that are in the map. First posted on arXiv 7 Apr 2025.

Builds on

41 of 45 references.

Closest ancestors · 12

Same direction, newest first

  1. 2025Process Reinforcement through Implicit Rewards3 Feb
  2. 2025Diverse Preference Optimization30 Jan
  3. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning22 Jan
  4. 2024Offline Reinforcement Learning for LLM Multi-Step Reasoning20 Dec
  5. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Training22 Nov
  6. 2024Enhancing Multi-Step Reasoning Abilities of Language Models through Direct Q-Function Optimization11 Oct
  7. 2024KTO: Model Alignment as Prospect Theoretic Optimization2 Feb
  8. 2023Let's Verify Step by Step31 May
  9. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  10. 2022Constitutional AI: Harmlessness from AI Feedback15 Dec
  11. 2022Solving math word problems with process- and outcome-based feedback25 Nov
  12. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 29

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2024The Llama 3 Herd of Modelscited by 14,192
  4. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  5. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  6. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  7. 2017Deep reinforcement learning from human preferencescited by 3,846
  8. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  9. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  10. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  11. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,928
  12. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,928
17 more
  1. 2022Competition-Level Code Generation with AlphaCodecited by 1,427
  2. 2024A Survey on LLM-as-a-Judgecited by 1,384
  3. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 951
  4. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 906
  5. 2022STaR: Bootstrapping Reasoning With Reasoningcited by 895
  6. 2021MuSiQue: Multihop Questions via Single-hop Question Compositioncited by 893
  7. 2024Gemma: Open Models Based on Gemini Research and Technologycited by 889
  8. 2025SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-trainingcited by 655
  9. 2023Reinforced Self-Training (ReST) for Language Modelingcited by 486
  10. 2023Scaling Relationship on Learning Mathematical Reasoning with Large Language Modelscited by 417
  11. 2023Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Modelscited by 325
  12. 2024RLEF: Grounding Code LLMs in Execution Feedback with Reinforcement Learningcited by 146
  13. 2024RL on Incorrect Synthetic Data Scales the Efficiency of LLM Math Reasoning by Eight-Foldcited by 99
  14. 2024Gecko: Versatile Text Embeddings Distilled from Large Language Modelscited by 81
  15. 2025CodeMonkeys: Scaling Test-Time Compute for Software Engineeringcited by 46
  16. 2020Answering Open-Domain Questions of Varying Reasoning Steps from Textcited by 43
  17. 2024Cofca: A Step-Wise Counterfactual Multi-hop QA benchmarkcited by 26

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 29; 2026: 16.

45 papers.

Sort:

By area:

  1. 2025The Landscape of Agentic Reinforcement Learning for LLMs: A Surveycited by 195
  2. 2025From LLM Reasoning to Autonomous AI Agents: A Comprehensive Reviewcited by 146
  3. 2025Deep Research Agents: A Systematic Examination And Roadmapcited by 122
  4. 2025Agent Learning via Early Experiencecited by 79
  5. 2025A Survey of LLM-based Deep Search Agents: Paradigm, Optimization, Evaluation, and Challengescited by 58
  6. 2025Visual Agentic Reinforcement Fine-Tuningcited by 53
  7. 2025SeRL: Self-Play Reinforcement Learning for Large Language Models with Limited Datacited by 50
  8. 2025ToolOrchestra: Elevating Intelligence via Efficient Model and Tool Orchestrationcited by 44
  9. 2025Reinforcement Learning Meets Large Language Models: A Survey of Advancements and Applications Across the LLM Lifecyclecited by 34
  10. 2025Agentic Plan Caching: Test-Time Memory for Fast and Cost-Efficient LLM Agentscited by 23
  11. 2025Get Experience from Practice: LLM Agents with Record & Replaycited by 22
  12. 2025A Comprehensive Survey on Reinforcement Learning-based Agentic Search: Foundations, Roles, Optimizations, Evaluations, and Applicationscited by 21
33 more
  1. 2025Sailing by the Stars: A Survey on Reward Models and Learning Strategies for Learning from Rewardscited by 14
  2. 2025Reinforcement Learning Foundations for Deep Research Systems: A Surveycited by 13
  3. 2026Toward Efficient Agents: Memory, Tool learning, and Planningcited by 11
  4. 2026Dynamic Skill Lifecycle Management for Agentic Reinforcement Learningcited by 10
  5. 2025Revisiting Test-Time Scaling: A Survey and a Diversity-Aware Method for Efficient Reasoningcited by 8
  6. 2025Beyond Pipelines: A Survey of the Paradigm Shift toward Model-Native Agentic AIcited by 8
  7. 2025EvoCoT: Overcoming the Exploration Bottleneck in Reinforcement Learningcited by 7
  8. 2025Self-Guided Function Calling in Large Language Models via Stepwise Experience Recallcited by 7
  9. 2025Enhancing Large Language Model Reasoning with Reward Models: An Analytical Surveycited by 7
  10. 2026SCALER:Synthetic Scalable Adaptive Learning Environment for Reasoningcited by 7
  11. 2025Achieving Olympiad-Level Geometry Large Language Model Agent via Complexity Boosting Reinforcement Learningcited by 6
  12. 2026Learning from Synthetic Data Improves Multi-hop Reasoningcited by 6
  13. 2026FinMCP-Bench: Benchmarking LLM Agents for Real-World Financial Tool Use under the Model Context Protocolcited by 6
  14. 2025AWPO: Enhancing Tool-Use of Large Language Models through Adaptive Integration of Reasoning Rewardscited by 5
  15. 2025Structural Entropy Guided Agent for Detecting and Repairing Knowledge Deficiencies in LLMscited by 4
  16. 2025MOTIF: Modular Thinking via Reinforcement Fine-tuning in LLMscited by 4
  17. 2025Even Small Reasoners Should Quote Their Sources: Introducing the Pleias-RAG Model Familycited by 3
  18. 2025No Labels, No Problem: Training Visual Reasoners with Multimodal Verifierscited by 3
  19. 2026Beyond Alignment: Expanding Reasoning Capacity via Manifold-Reshaping Policy Optimizationcited by 3
  20. 2026A Deep Dive into Scaling RL for Code Generation with Synthetic Data and Curriculacited by 3
  21. 2026S3-R1: Learning to Retrieve and Answer Step-by-Step with Synthetic Datacited by 2
  22. 2026DocArena: Turning Raw Documents into Controllable Training Environments for Document Search Agentscited by 2
  23. 2025Physics-Based Benchmarking Metrics for Multimodal Synthetic Imagescited by 1
  24. 2026TTVS: Boosting Self-Exploring Reinforcement Learning via Test-time Variational Synthesiscited by 1
  25. 2026Fine-Tuning Small Reasoning Models for Quantum Field Theorycited by 1
  26. 2026Process Supervision of Confidence Margin for Calibrated LLM Reasoningcited by 1
  27. 2026Masked Diffusion Language Models are Strong and Steerable Text-Based World Models for Agentic RLcited by 1
  28. 2025RACE-Align: Retrieval-Augmented and Chain-of-Thought Enhanced Preference Alignment for Large Language Modelscited by 0
  29. 2025Step-wise Policy for Rare-tool Knowledge (SPaRK): Offline RL that Drives Diverse Tool Use in LLMscited by 0
  30. 2025SpatialTraceGen: High-Fidelity Traces for Efficient VLM Spatial Reasoning Distillationcited by 0
  31. 2026PyraMathBench: Evaluating and Improving Mathematical Capability in Large Language Modelscited by 0
  32. 2026Multi-turn RL with Structural and Performance Aware Rewards for CUDA Kernel Generationcited by 0
  33. 2026PhantomEnvironments: Training LLM Agents in Fictional Worldscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Mid-Training with Self-Generated Data Improves Reinforcement Learning in Language Models
  2. 2025Tool-R1: Sample-Efficient Reinforcement Learning for Agentic Tool Use
  3. 2025Synthetic Data RL: Task Definition Is All You Need
  4. 2022Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimization
  5. 2026A Survey of Reinforcement Learning for Large Language Models under Data Scarcity: Challenges and Solutions
  6. 2024TeaMs-RL: Teaching LLMs to Generate Better Instruction Datasets via Reinforcement Learning
  7. 2025A Sober Look at Progress in Language Model Reasoning: Pitfalls and Paths to Reproducibility
  8. 2025Effective Reinforcement Learning for Reasoning in Language Models