mlsanityData to 7 Oct 2026

A Survey of Reinforcement Learning for Large Language Models under Data Scarcity: Challenges and Solutions

arXiv 2604.17312 ↗First posted 19 Apr 2026cs.LG cs.AIcited by 0

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Reinforcement learning (RL) has emerged as a powerful post-training paradigm for enhancing the reasoning capabilities of large language models (LLMs). However, reinforcement learning for LLMs faces substantial data scarcity challenges, including the limited availability of high-quality external supervision and the constrained volume of model-generated experience. These limitations make data-efficient reinforcement learning a critical research direction. In this survey, we present the first systematic review of reinforcement learning for LLMs under data scarcity. We propose a bottom-up hierarchical framework built around three complementary perspectives: the data-centric perspective, the training-centric perspective, and the framework-centric perspective. We develop a taxonomy of existing methods, summarize representative approaches in each category, and analyze their strengths and limitations. Our taxonomy aims to provide a clear conceptual foundation for understanding the design space of data-efficient RL for LLMs and to guide researchers working in this emerging area. We hope this survey offers a comprehensive roadmap for future research and inspires new directions toward more efficient and scalable reinforcement learning post-training for LLMs.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 99 of its 129 references that are in the map. First posted on arXiv 19 Apr 2026.

Builds on

99 of 129 references.

Closest ancestors · 38

Same direction, newest first

  1. 2026MAESTRO: Meta-learning Adaptive Estimation of Scalarization Trade-offs for Reward Optimization12 Jan
  2. 2025Generative Adversarial Reasoner: Enhancing LLM Reasoning with Adversarial Reinforcement Learning18 Dec
  3. 2025Better LLM Reasoning via Dual-Play14 Nov
  4. 2025Efficient Reinforcement Learning for Large Language Models with Intrinsic Exploration2 Nov
  5. 2025Towards Flash Thinking via Decoupled Advantage Policy Optimization17 Oct
  6. 2025LSPO: Length-aware Dynamic Sampling for Policy Optimization in LLM Reasoning1 Oct
  7. 2025Sample-efficient LLM Optimization with Reset Replay8 Aug
  8. 2025Can Prompt Difficulty be Online Predicted for Accelerating RL Finetuning of Reasoning Models?7 Jul
  9. 2025Spurious Rewards: Rethinking Training Signals in RLVR12 Jun
  10. 2025SPEED-RL: Faster Training of Reasoning Models via Online Curriculum Learning10 Jun
  11. 2025SwS: Self-aware Weakness-driven Problem Synthesis in Reinforcement Learning for LLM Reasoning10 Jun
  12. 2025Consistent Paths Lead to Truth: Self-Rewarding Reinforcement Learning for LLM Reasoning10 Jun
26 more
  1. 2025Confidence Is All You Need: Few-Shot RL Fine-Tuning of Language Models5 Jun
  2. 2025Improving Data Efficiency for LLM Reinforcement Fine-tuning Through Difficulty-targeted Online Data Selection and Rollout Replay5 Jun
  3. 2025Reshaping Reasoning in LLMs: A Theoretical Analysis of RL Training Dynamics through Pattern Selection5 Jun
  4. 2025Act Only When It Pays: Efficient Reinforcement Learning for LLM Reasoning via Selective Rollouts2 Jun
  5. 2025Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning2 Jun
  6. 2025Maximizing Confidence Alone Improves Reasoning28 May
  7. 2025Can Large Reasoning Models Self-Train?27 May
  8. 2025SynLogic: Synthesizing Verifiable Reasoning Data at Scale for Learning Logical Reasoning and Beyond26 May
  9. 2025Learning to Reason without External Rewards26 May
  10. 2025SeRL: Self-Play Reinforcement Learning for Large Language Models with Limited Data25 May
  11. 2025The Unreasonable Effectiveness of Entropy Minimization in LLM Reasoning21 May
  12. 2025Do Not Let Low-Probability Tokens Over-Dominate in RL for LLMs19 May
  13. 2025UFO-RL: Uncertainty-Focused Optimization for Efficient Reinforcement Learning Data Selection18 May
  14. 2025Learning to Think: Information-Theoretic Reinforcement Fine-Tuning for LLMs15 May
  15. 2025S-GRPO: Early Exit via Reinforcement Learning in Reasoning Models12 May
  16. 2025Absolute Zero: Reinforced Self-play Reasoning with Zero Data6 May
  17. 2025Reinforcement Learning for Reasoning in Large Language Models with One Training Example29 Apr
  18. 2025Not All Rollouts are Useful: Down-Sampling Rollouts in LLM Reinforcement Learning18 Apr
  19. 2025Online Difficulty Filtering for Reasoning Oriented Reinforcement Learning4 Apr
  20. 2025Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Model31 Mar
  21. 2025CPPO: Accelerating the Training of Group Relative Policy Optimization-Based Reasoning Models28 Mar
  22. 2025SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wild24 Mar
  23. 2025LIMR: Less is More for RL Scaling17 Feb
  24. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMs22 Jan
  25. 2024AlphaMath Almost Zero: Process Supervision without Process6 May
  26. 2024KTO: Model Alignment as Prospect Theoretic Optimization2 Feb

Other references · 61

  1. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  2. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  3. 2020Learning to summarize from human feedbackcited by 2,562
  4. 2025Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learningcited by 1,371
  5. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  6. 2024DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligencecited by 1,154
  7. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 825
  8. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 630
  9. 2023Reinforced Self-Training (ReST) for Language Modelingcited by 486
  10. 2024Training Language Models to Self-Correct via Reinforcement Learningcited by 405
  11. 2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Trainingcited by 399
  12. 2023UltraFeedback: Boosting Language Models with Scaled AI Feedbackcited by 394
38 more
  1. 2025rStar-Math: Small LLMs Can Master Math Reasoning with Self-Evolved Deep Thinkingcited by 308
  2. 2023Alphazero-like Tree-Search can Guide Large Language Model Decoding and Trainingcited by 291
  3. 2024Self-Play Preference Optimization for Language Model Alignmentcited by 228
  4. 2025R-Zero: Self-Evolving Reasoning LLM from Zero Datacited by 210
  5. 2025A Comprehensive Survey of Self-Evolving AI Agents: A New Paradigm Bridging Foundation Models and Lifelong Agentic Systemscited by 177
  6. 2017Intrinsic Motivation and Automatic Curricula via Asymmetric Self-Playcited by 139
  7. 2025REASONING GYM: Reasoning Environments for Reinforcement Learning with Verifiable Rewardscited by 121
  8. 2024A Note on Shumailov et al. (2024): `AI Models Collapse When Trained on Recursively Generated Data'cited by 114
  9. 2024Self-playing Adversarial Language Game Enhances LLM Reasoningcited by 80
  10. 2025SPICE: Self-Play In Corpus Environments Improves Reasoningcited by 78
  11. 2021Asymmetric self-play for automatic goal discovery in robotic manipulationcited by 71
  12. 2025Self-rewarding correction for mathematical reasoningcited by 67
  13. 2025Multi-Agent Evolve: LLM Self-Improve through Co-evolutioncited by 58
  14. 2024CodeUltraFeedback: An LLM-as-a-Judge Dataset for Aligning Large Language Models to Coding Preferencescited by 56
  15. 2025Enigmata: Scaling Logical Reasoning in Large Language Models with Synthetic Verifiable Puzzlescited by 53
  16. 2024Beyond Chain-of-Thought: A Survey of Chain-of-X Paradigms for LLMscited by 48
  17. 2025ZeroGUI: Automating Online GUI Learning at Zero Human Costcited by 44
  18. 2025Evolving Language Models without Labels: Majority Drives Selection, Novelty Promotes Variationcited by 44
  19. 2025VisPlay: Self-Evolving Vision-Language Models from Imagescited by 42
  20. 2024Prompting Large Language Models to Tackle the Full Software Development Lifecycle: A Case Studycited by 40
  21. 2025SPC: Evolving Self-Play Critic via Adversarial Games for LLM Reasoningcited by 40
  22. 2025Vision-Zero: Scalable VLM Self-Improvement via Strategic Gamified Self-Playcited by 37
  23. 2025Toward Training Superintelligent Software Agents through Self-Play SWE-RLcited by 37
  24. 2025Search Self-play: Pushing the Frontier of Agent Capability without Supervisioncited by 31
  25. 2025Beyond Correctness: Harmonizing Process and Outcome Rewards through RL Trainingcited by 29
  26. 2025SPELL: Self-Play Reinforcement Learning for Evolving Long-Context Language Modelscited by 27
  27. 2025CoMAS: Co-Evolving Multi-Agent Systems via Interaction Rewardscited by 27
  28. 2024Model Surgery: Modulating LLM's Behavior Via Simple Parameter Editingcited by 23
  29. 2025Pareto Multi-Objective Alignment for Language Modelscited by 20
  30. 2025Inpainting-Guided Policy Optimization for Diffusion Large Language Modelscited by 16
  31. 2025LoopTool: Closing the Data-Training Loop for Robust LLM Tool Callscited by 16
  32. 2025Toward Efficient Exploration by Large Language Model Agentscited by 15
  33. 2025SSR-Zero: Simple Self-Rewarding Reinforcement Learning for Machine Translationcited by 13
  34. 2025Angles Don't Lie: Unlocking Training-Efficient RL Through the Model's Own Signalscited by 13
  35. 2025SynthRL: Scaling Visual Reasoning with Verifiable Data Synthesiscited by 12
  36. 2025BOTS: A Unified Framework for Bayesian Online Task Selection in LLM Reinforcement Finetuningcited by 12
  37. 2025Surrogate Signals from Format and Length: Reinforcement Learning for Solving Mathematical Problems without Ground Truth Answerscited by 11
  38. 2025Propose, Solve, Verify: Self-Play Through Formal Verificationcited by 11

50 most cited of 61 All 61 (JSON)

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Learning at the Right Pace: Adaptive Data Scheduling Improves LLM Reinforcement Learning
  2. 2025A Survey of Reinforcement Learning for Large Reasoning Models
  3. 2024TeaMs-RL: Teaching LLMs to Generate Better Instruction Datasets via Reinforcement Learning
  4. 2025Training Language Models to Reason Efficiently
  5. 2025Behavior Injection: Preparing Language Models for Reinforcement Learning
  6. 2025TTRL: Test-Time Reinforcement Learning
  7. 2025Enhancing Efficiency and Exploration in Reinforcement Learning for LLMs
  8. 2025Diversity-Aware Policy Optimization for Large Language Model Reasoning