mlsanityData to 7 Oct 2026

Learning from Synthetic Data Improves Multi-hop Reasoning

arXiv 2603.02091 ↗First posted 2 Mar 2026cs.LG cs.AI cs.CLcited by 6

On the map: LLM reasoning → Reinforcement learning for reasoning

also in 1 more
Abstract (arXiv)

Reinforcement Learning (RL) has been shown to significantly boost reasoning capabilities of large language models (LLMs) in math, coding, and multi-hop reasoning tasks. However, RL fine-tuning requires abundant high-quality verifiable data, often sourced from human annotations, generated from frontier LLMs, or scored by LLM-based verifiers. All three have considerable limitations: human-annotated datasets are small and expensive to curate, LLM-generated data is hallucination-prone and costly, and LLM-based verifiers are inaccurate and slow. In this work, we investigate a cheaper alternative: RL fine-tuning on rule-generated synthetic data for multi-hop reasoning tasks. We discover that LLMs fine-tuned on synthetic data perform significantly better on popular real-world question-answering benchmarks, despite the synthetic data containing only fictional knowledge. On stratifying performance by question difficulty, we find that synthetic data teaches LLMs to compose knowledge -- a fundamental and generalizable reasoning skill. Our work highlights rule-generated synthetic reasoning data as a free and scalable resource to improve LLM reasoning capabilities.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 46 of its 89 references that are in the map. First posted on arXiv 2 Mar 2026.

Builds on

46 of 89 references.

Closest ancestors · 18

Same direction, newest first

  1. 2025SkillFactory: Self-Distillation For Learning Cognitive Behaviors3 Dec
  2. 2025Sample More to Think Less: Group Filtered Policy Optimization for Concise Reasoning13 Aug
  3. 2025Reasoning or Memorization? Unreliable Results of Reinforcement Learning Due to Data Contamination14 Jul
  4. 2025Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMs17 Jun
  5. 2025Spurious Rewards: Rethinking Training Signals in RLVR12 Jun
  6. 2025ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Models30 May
  7. 2025Accelerating RL for LLM Reasoning with Optimal Advantage Regression27 May
  8. 2025Learning to Reason without External Rewards26 May
  9. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?18 Apr
  10. 2025Synthetic Data Generation & Multi-Step RL for Reasoning & Tool Use7 Apr
  11. 2025Crossing the Reward Bridge: Expanding RL with Verifiable Rewards Across Diverse Domains31 Mar
  12. 2025Understanding R1-Zero-Like Training: A Critical Perspective26 Mar
6 more
  1. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  2. 2025Cognitive Behaviors that Enable Self-Improving Reasoners, or, Four Habits of Highly Effective STaRs3 Mar
  3. 2025REINFORCE++: Stabilizing Critic-Free Policy Optimization with Global Advantage Normalization4 Jan
  4. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Training22 Nov
  5. 2022Constitutional AI: Harmlessness from AI Feedback15 Dec
  6. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 28

  1. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  3. 2025Qwen3 Technical Reportcited by 8,195
  4. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  5. 2024τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domainscited by 1,280
  6. 2024GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Modelscited by 533
  7. 2025The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexitycited by 298
  8. 2023FireAct: Toward Language Agent Fine-tuningcited by 247
  9. 2025ARC-AGI-2: A New Challenge for Frontier AI Reasoning Systemscited by 171
  10. 2025Phi-4-reasoning Technical Reportcited by 146
  11. 2025REASONING GYM: Reasoning Environments for Reinforcement Learning with Verifiable Rewardscited by 121
  12. 2024On Memorization of Large Language Models in Logical Reasoningcited by 114
16 more
  1. 2025Towards System 2 Reasoning in LLMs: Learning How to Think With Meta Chain-of-Thoughtcited by 108
  2. 2024Best Practices and Lessons Learned on Synthetic Datacited by 104
  3. 2023ARB: Advanced Reasoning Benchmark for Large Language Modelscited by 60
  4. 2025Reasoning to Learn from Latent Thoughtscited by 43
  5. 2025From f(x) and g(x) to f(g(x)): LLMs Learn New Skills in RL by Composing Old Onescited by 40
  6. 2025Physics of Language Models: Part 4.1, Architecture Design and the Magic of Canon Layerscited by 32
  7. 2025NaturalThoughts: Selecting and Distilling Reasoning Traces for General Reasoning Taskscited by 29
  8. 2025Interleaved Reasoning for Large Language Models via Reinforcement Learningcited by 21
  9. 2025How Do LLMs Perform Two-Hop Reasoning in Context?cited by 20
  10. 2024From Artificial Needles to Real Haystacks: Improving Retrieval Capabilities in LLMs by Finetuning on Synthetic Datacited by 18
  11. 2025Language Models Improve When Pretraining Data Matches Target Taskscited by 18
  12. 2025PhantomWiki: On-Demand Datasets for Reasoning and Retrieval Evaluationcited by 14
  13. 2025General Intelligence Requires Reward-based Pretrainingcited by 12
  14. 2024Understanding Synthetic Context Extension via Retrieval Headscited by 10
  15. 2025SynthWorlds: Controlled Parallel Worlds for Disentangling Reasoning and Knowledge in Language Modelscited by 5
  16. 2025π-CoT: Prolog-Initialized Chain-of-Thought Prompting for Multi-Hop Question-Answeringcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 6.

6 papers.

  1. 2026Scaling Multi-Hop Training Data via Graph-Constrained Path Selectioncited by 1
  2. 2026Do Language Models Need Sleep? Offline Recurrence for Improved Online Inferencecited by 0
  3. 2026MultiSynt/MT: Trillion-Token Multi-Parallel Pre-Training Data Translated Across 36 Languagescited by 0
  4. 2026Video-HopChain: Multi-Hop Questions and Confidence-Gated Exploration for Video Reasoning Modelscited by 0
  5. 2026Lasting Effects of Abstract Pretraining Beyond Perplexitycited by 0
  6. 2026PhantomEnvironments: Training LLM Agents in Fictional Worldscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Enhancing Logical Reasoning in Large Language Models through Graph-based Synthetic Data
  2. 2025Loong: Synthesize Long Chain-of-Thoughts at Scale through Verifiers
  3. 2025Logic-RL: Unleashing LLM Reasoning with Rule-Based Reinforcement Learning
  4. 2025Synthetic Data Generation Using Large Language Models: Advances in Text and Code
  5. 2025General-Reasoner: Advancing LLM Reasoning Across All Domains
  6. 2026CHIMERA: Compact Synthetic Data for Generalizable LLM Reasoning
  7. 2025A Survey of Reinforcement Learning for Large Reasoning Models
  8. 2026ORACLE: Optimizing Reasoning Abilities of Large Language Models via Constraint-Led Synthetic Data Elicitation