mlsanityData to 7 Oct 2026

Towards High Data Efficiency in Reinforcement Learning with Verifiable Reward

arXiv 2509.01321 ↗First posted 1 Sep 2025cs.LG cs.CLcited by 16

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Recent advances in large reasoning models have leveraged reinforcement learning with verifiable rewards (RLVR) to improve reasoning capabilities. However, scaling these methods typically requires extensive rollout computation and large datasets, leading to high training costs and low data efficiency. To mitigate this issue, we propose DEPO, a Data-Efficient Policy Optimization pipeline that combines optimized strategies for both offline and online data selection. In the offline phase, we curate a high-quality subset of training samples based on diversity, influence, and appropriate difficulty. During online RLVR training, we introduce a sample-level explorability metric to dynamically filter samples with low exploration potential, thereby reducing substantial rollout computational costs. Furthermore, we incorporate a replay mechanism for under-explored samples to ensure adequate training, which enhances the model's final convergence performance. Experiments across five reasoning benchmarks show that DEPO consistently outperforms existing methods in both offline and online data selection scenarios. Notably, using only 20% of the training data, our approach achieves a 1.85 times speed-up on AIME24 and a 1.66 times speed-up on AIME25 compared to GRPO trained on the full dataset.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 29 of its 36 references that are in the map. First posted on arXiv 1 Sep 2025.

Builds on

29 of 36 references.

Closest ancestors · 11

Same direction, newest first

  1. 2025Part I: Tricks or Traps? A Deep Dive into RL for LLM Reasoning11 Aug
  2. 2025Reasoning with Exploration: An Entropy Perspective17 Jun
  3. 2025Act Only When It Pays: Efficient Reinforcement Learning for LLM Reasoning via Selective Rollouts2 Jun
  4. 2025Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning2 Jun
  5. 2025Reinforcement Learning for Reasoning in Large Language Models with One Training Example29 Apr
  6. 2025VAPO: Efficient and Reliable Reinforcement Learning for Advanced Reasoning Tasks7 Apr
  7. 2025Understanding R1-Zero-Like Training: A Critical Perspective26 Mar
  8. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  9. 2025LIMR: Less is More for RL Scaling17 Feb
  10. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning22 Jan
  11. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMs22 Jan

Other references · 18

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  4. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  5. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
  6. 2023A Survey of Large Language Modelscited by 2,537
  7. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 1,980
  8. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 1,906
  9. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,057
  10. 2023PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallelcited by 641
  11. 2023In-Context Learning Creates Task Vectorscited by 302
  12. 2023In-context Vectors: Making In Context Learning More Effective and Controllable Through Latent Space Steeringcited by 240
6 more
  1. 2024Improving Instruction-Following in Language Models through Activation Steeringcited by 149
  2. 2023The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Datasetcited by 148
  3. 2024Cut the Crap: An Economical Communication Pipeline for LLM-based Multi-Agent Systemscited by 129
  4. 2024Perplexed by Perplexity: Perplexity-Based Data Pruning With Small Reference Modelscited by 83
  5. 2025Unlocking General Long Chain-of-Thought Reasoning Capabilities of Large Language Models via Representation Engineeringcited by 34
  6. 2024HARP: A challenging human-annotated math reasoning benchmarkcited by 25

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 3; 2026: 13.

16 papers.

Sort:

By area:

  1. 2025Rethinking Sample Polarity in Reinforcement Learning with Verifiable Rewardscited by 19
  2. 2025Beyond Majority Voting: Towards Fine-grained and More Reliable Reward Signal for Test-Time Reinforcement Learningcited by 11
  3. 2026RLVR without Ineffective Samples: Group Prioritized Off-Policy Optimization for LLM Reasoningcited by 7
  4. 2025Detecting Data Contamination from Reinforcement Learning Post-training for Large Language Modelscited by 4
  5. 2026EDCO: Dynamic Curriculum Orchestration for Domain-specific Large Language Model Fine-tuningcited by 2
  6. 2026DeReason: A Difficulty-Aware Curriculum Improves Decoupled SFT-then-RL Training for General Reasoningcited by 2
  7. 2026Low-rank Optimization Trajectories Modeling for LLM RLVR Accelerationcited by 2
  8. 2026DARE: Difficulty-Adaptive Reinforcement Learning with Co-Evolved Difficulty Estimationcited by 2
  9. 2026Group Pattern Selection Optimization: Let LRMs Pick the Right Pattern for Reasoningcited by 1
  10. 2026IRDS: Interpretable RLVR Data Selection via Verifier-Coupled Sparse Autoencoder Coveragecited by 1
  11. 2026Mechanistically Interpreting the Role of Sample Difficulty in RLVR for LLMscited by 1
  12. 2026Smart Picks in the Dark: Towards Efficient RLVR for Reasoning via Tracing Metacognitive Pivotscited by 1
4 more
  1. 2026Demystifying Design Choices of Reinforcement Fine-tuning: A Batched Contextual Bandit Learning Perspectivecited by 0
  2. 2026Reasoning Arena: Trace Tournaments When Verifiable Rewards Fall Shortcited by 0
  3. 2026Unlocking the Unsolvable: Teacher-Guided Curriculum for Data-Efficient RLVRcited by 0
  4. 2026Unlearnable, or Unmeasured? On the Reliability of Difficulty Labels in RLVRcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026DataFlex-RL: An Evaluation Platform for RLVR Data Policies
  2. 2026Resource-Efficient Reinforcement for Reasoning Large Language Models via Dynamic One-Shot Policy Refinement
  3. 2026Single-Rollout Hidden-State Dynamics for Training-Free RLVR Data Selection
  4. 2025Explore Data Left Behind in Reinforcement Learning for Reasoning Language Models
  5. 2025Efficient Reinforcement Learning for Large Language Models with Intrinsic Exploration
  6. 2026Learning as Reasoning Unfolds: Progressive Rollout Allocation for Efficient Reinforcement Learning
  7. 2025DRPO: Efficient Reasoning via Decoupled Reward Policy Optimization
  8. 2025Online Difficulty Filtering for Reasoning Oriented Reinforcement Learning