mlsanityData to 7 Oct 2026

RL on Incorrect Synthetic Data Scales the Efficiency of LLM Math Reasoning by Eight-Fold

arXiv 2406.14532 ↗First posted 20 Jun 2024cs.LG cs.CLcited by 99

On the map: LLM reasoning → Mathematical reasoning

also in 1 more
Abstract (arXiv)

Training on model-generated synthetic data is a promising approach for finetuning LLMs, but it remains unclear when it helps or hurts. In this paper, we investigate this question for math reasoning via an empirical study, followed by building a conceptual understanding of our observations. First, we find that while the typical approach of finetuning a model on synthetic correct or positive problem-solution pairs generated by capable models offers modest performance gains, sampling more correct solutions from the finetuned learner itself followed by subsequent fine-tuning on this self-generated data doubles the efficiency of the same synthetic problems. At the same time, training on model-generated positives can amplify various spurious correlations, resulting in flat or even inverse scaling trends as the amount of data increases. Surprisingly, we find that several of these issues can be addressed if we also utilize negative responses, i.e., model-generated responses that are deemed incorrect by a final answer verifier. Crucially, these negatives must be constructed such that the training can appropriately recover the utility or advantage of each intermediate step in the negative response. With this per-step scheme, we are able to attain consistent gains over only positive data, attaining performance similar to amplifying the amount of synthetic data by 8 ×. We show that training on per-step negatives can help to unlearn spurious correlations in the positive data, and is equivalent to advantage-weighted reinforcement learning (RL), implying that it inherits robustness benefits of RL over imitating positive data alone.

Key facts. Cited by 99 papers in the map (data through 7 Oct 2026). Placed in Mathematical reasoning, one of 17 directions in LLM reasoning. Builds on 49 of its 69 references that are in the map. First posted on arXiv 20 Jun 2024.

Builds on

49 of 69 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024Iterative Reasoning Preference Optimization30 Apr
  2. 2024Common 7B Language Models Already Possess Strong Math Capabilities7 Mar
  3. 2023Scaling Relationship on Learning Mathematical Reasoning with Large Language Models3 Aug

Other references · 46

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  6. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  7. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  8. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  9. 2022STaR: Bootstrapping Reasoning With Reasoningcited by 895
  10. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
  11. 2024Self-Rewarding Language Modelscited by 654
  12. 2024DeepSeek LLM: Scaling Open-Source Language Models with Longtermismcited by 621
34 more
  1. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 552
  2. 2023Faith and Fate: Limits of Transformers on Compositionalitycited by 496
  3. 2024Negative Preference Optimization: From Catastrophic Collapse to Effective Unlearningcited by 484
  4. 2024Improve Mathematical Reasoning in Language Models by Automated Process Supervisioncited by 426
  5. 2024Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translationcited by 390
  6. 2023The Curse of Recursion: Training on Generated Data Makes Models Forgetcited by 343
  7. 2023Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Modelscited by 325
  8. 2019Causal Confusion in Imitation Learningcited by 286
  9. 2023Self-Consuming Generative Models Go MADcited by 239
  10. 2024Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positivecited by 239
  11. 2023How Abilities in Large Language Models are Affected by Supervised Fine-tuning Data Compositioncited by 222
  12. 2023Nash Learning from Human Feedbackcited by 220
  13. 2022Memorization Without Overfitting: Analyzing the Training Dynamics of Large Language Modelscited by 218
  14. 2024V-STaR: Training Verifiers for Self-Taught Reasonerscited by 214
  15. 2024Advancing LLM Reasoning Generalists with Preference Treescited by 191
  16. 2024Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Datacited by 184
  17. 2020Large-Scale Methods for Distributionally Robust Optimizationcited by 179
  18. 2023Embers of Autoregression: Understanding Large Language Models Through the Problem They are Trained to Solvecited by 175
  19. 2024The pitfalls of next-token predictioncited by 156
  20. 2024Is Model Collapse Inevitable? Breaking the Curse of Recursion by Accumulating Real and Synthetic Datacited by 143
  21. 2024A Minimaximalist Approach to Reinforcement Learning from Human Feedbackcited by 141
  22. 2022Calibrating Sequence likelihood Improves Conditional Language Generationcited by 121
  23. 2023Is RLHF More Difficult than Standard RL?cited by 96
  24. 2022When Should We Prefer Offline Reinforcement Learning Over Behavioral Cloning?cited by 86
  25. 2024Model Collapse Demystified: The Case of Regressioncited by 80
  26. 2023Evaluating Cognitive Maps and Planning in Large Language Models with CogEvalcited by 77
  27. 2024How Bad is Training on Synthetic Data? A Statistical Analysis of Language Model Collapsecited by 75
  28. 2023Pairwise Proximal Policy Optimization: Harnessing Relative Feedback for LLM Alignmentcited by 57
  29. 2024Fairness Feedback Loops: Training on Synthetic Data Amplifies Biascited by 55
  30. 2024A Critical Evaluation of AI Feedback for Aligning Large Language Modelscited by 36
  31. 2024Insights into Alignment: Evaluating DPO and its Variants Across Multiple Taskscited by 35
  32. 2023Adversarial Preference Optimization: Enhancing Your Alignment via RM-LLM Gamecited by 27
  33. 2023Bitrate-Constrained DRO: Beyond Worst Case Robustness To Unknown Group Shiftscited by 9
  34. 2023Exploration with Principles for Diverse AI Supervisioncited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 26; 2025: 52; 2026: 21.

99 papers.

Sort:

By area:

  1. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,928
  2. 2024Generative Verifiers: Reward Modeling as Next-Token Predictioncited by 465
  3. 2024Rewarding Progress: Scaling Automated Process Verifiers for LLM Reasoningcited by 267
  4. 2024Agent Q: Advanced Reasoning and Learning for Autonomous AI Agentscited by 201
  5. 2024VinePPO: Refining Credit Assignment in RL Training of LLMscited by 168
  6. 2024Improve Vision Language Model Chain-of-thought Reasoningcited by 135
  7. 2025Can 1B LLM Surpass 405B LLM? Rethinking Compute-Optimal Test-Time Scalingcited by 135
  8. 2025Optimizing Test-Time Compute via Meta Reinforcement Fine-Tuningcited by 121
  9. 2025Towards System 2 Reasoning in LLMs: Learning How to Think With Meta Chain-of-Thoughtcited by 108
  10. 2025Scaling Test-Time Compute Without Verification or RL is Suboptimalcited by 82
  11. 2024Process Reward Model with Q-Value Rankingscited by 78
  12. 2024MALT: Improving Reasoning with Multi-Agent LLM Trainingcited by 71
38 more
  1. 2025The Best Instruction-Tuning Data are Those That Fitcited by 48
  2. 2025Synthetic Data Generation & Multi-Step RL for Reasoning & Tool Usecited by 45
  3. 2024From Multimodal LLMs to Generalist Embodied Agents: Methods and Lessonscited by 42
  4. 2025Language Self-Play For Data-Free Trainingcited by 42
  5. 2025S2R: Teaching LLMs to Self-verify and Self-correct via Reinforcement Learningcited by 37
  6. 2024Self-Improving Diffusion Models with Synthetic Datacited by 34
  7. 2024Surveying the Effects of Quality, Diversity, and Complexity in Synthetic Data From Large Language Modelscited by 31
  8. 2025InSTA: Towards Internet-Scale Training For Agentscited by 31
  9. 2025MathFusion: Enhancing Mathematical Problem-solving of LLM through Instruction Fusioncited by 30
  10. 2024What Makes Large Language Models Reason in (Multi-Turn) Code Generation?cited by 29
  11. 2025Recognition through Reasoning: Reinforcing Image Geo-localization with Large Vision-Language Modelscited by 27
  12. 2025Exploiting Tree Structure for Credit Assignment in RL Training of LLMscited by 27
  13. 2024Selective Preference Optimization via Token-Level Reward Function Estimationcited by 25
  14. 2024HARP: A challenging human-annotated math reasoning benchmarkcited by 25
  15. 2024What Do Learning Dynamics Reveal About Generalization in LLM Reasoning?cited by 22
  16. 2024Policy Filtration for RLHF to Mitigate Noise in Reward Modelscited by 21
  17. 2024To the Globe (TTG): Towards Language-Driven Guaranteed Travel Planningcited by 21
  18. 2024Scalable Reinforcement Post-Training Beyond Static Human Prompts: Evolving Alignment via Asymmetric Self-Playcited by 21
  19. 2025QLASS: Boosting Language Agent Inference via Q-Guided Stepwise Searchcited by 20
  20. 2025Population-Aligned Persona Generation for LLM-based Social Simulationcited by 20
  21. 2025Stepwise Guided Policy Optimization: Coloring your Incorrect Reasoning in GRPOcited by 18
  22. 2026Reuse your FLOPs: Scaling RL on Hard Problems by Conditioning on Very Off-Policy Prefixescited by 16
  23. 2024Towards a Theoretical Understanding of Synthetic Data in LLM Post-Training: A Reverse-Bottleneck Perspectivecited by 15
  24. 2025Synthetic Data RL: Task Definition Is All You Needcited by 15
  25. 2025Every Rollout Counts: Optimal Resource Allocation for Efficient Test-Time Scalingcited by 15
  26. 2024Balancing Cost and Effectiveness of Synthetic Data Generation Strategies for LLMscited by 14
  27. 2025Generative AI Act II: Test Time Scaling Drives Cognition Engineeringcited by 14
  28. 2025LongPerceptualThoughts: Distilling System-2 Reasoning for System-1 Perceptioncited by 14
  29. 2026InT: Self-Proposed Interventions Enable Credit Assignment in LLM Reasoningcited by 14
  30. 2024Mitigating Tail Narrowing in LLM Self-Improvement via Socratic-Guided Samplingcited by 13
  31. 2025How Much Backtracking is Enough? Exploring the Interplay of SFT and RL in Enhancing LLM Reasoningcited by 13
  32. 2025ExPO: Unlocking Hard Reasoning with Self-Explanation-Guided Reinforcement Learningcited by 13
  33. 2024CPL: Critical Plan Step Learning Boosts LLM Generalization in Reasoning Taskscited by 11
  34. 2025Structural Reward Model: Enhancing Interpretability, Efficiency, and Scalability in Reward Modelingcited by 11
  35. 2024A Common Pitfall of Margin-based Language Model Alignment: Gradient Entanglementcited by 10
  36. 2025CEC-Zero: Zero-Supervision Character Error Correction with Self-Generated Rewardscited by 10
  37. 2026What If Consensus Lies? Selective-Complementary Reinforcement Learning at Test Timecited by 10
  38. 2024Derailer-Rerailer: Adaptive Verification for Efficient and Reliable Language Model Reasoningcited by 9

50 most cited of 99 All 99 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025FLAMES: Improving LLM Math Reasoning via a Fine-Grained Analysis of the Data Synthesis Pipeline
  2. 2023Turning Dust into Gold: Distilling Complex Reasoning Capabilities from LLMs by Leveraging Negative Data
  3. 2024Physics of Language Models: Part 2.2, How to Learn From Mistakes on Grade-School Math Problems
  4. 2025Examining False Positives under Inference Scaling for Mathematical Reasoning
  5. 2025LEMMA: Learning from Errors for MatheMatical Advancement in LLMs
  6. 2025Do LLMs Overthink Basic Math Reasoning? Benchmarking the Accuracy-Efficiency Tradeoff in Language Models
  7. 2025SPARQ: Synthetic Problem Generation for Reasoning via Quality-Diversity Algorithms
  8. 2023Learning From Mistakes Makes LLM Better Reasoner