# Prioritized Replay for RL Post-training

arXiv: https://arxiv.org/abs/2601.02648 · HTML: https://mlsanity.com/paper/2601.02648/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 24 references that are in the map.
- First posted on arXiv 6 Jan 2026.

## Builds on (most cited first, 14 of 14)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [Qwen2 Technical Report](https://mlsanity.com/paper/2407.10671/) (2024)
- [HybridFlow: A Flexible and Efficient RLHF Framework](https://mlsanity.com/paper/2409.19256/) (2024)
- [Prioritized Experience Replay](https://mlsanity.com/paper/1511.05952/) (2015)
- [Rainbow: Combining Improvements in Deep Reinforcement Learning](https://mlsanity.com/paper/1710.02298/) (2017)
- [Curriculum Learning for Reinforcement Learning Domains: A Framework and Survey](https://mlsanity.com/paper/2003.04960/) (2020)
- [Self-Evolving Curriculum for LLM Reasoning](https://mlsanity.com/paper/2505.14970/) (2025)
- [Concise Reasoning via Reinforcement Learning](https://mlsanity.com/paper/2504.05185/) (2025)
- [Revisiting Reinforcement Learning for LLM Reasoning from A Cross-Domain Perspective](https://mlsanity.com/paper/2506.14965/) (2025)
- [Improving Data Efficiency for LLM Reinforcement Fine-tuning Through Difficulty-targeted Online Data Selection and Rollout Replay](https://mlsanity.com/paper/2506.05316/) (2025)
- [Investigating the Interplay of Prioritized Replay and Generalization](https://mlsanity.com/paper/2407.09702/) (2024)

## Built on by (most cited first, 4 of 4)

- [Freshness-Aware Prioritized Experience Replay for LLM/VLM Reinforcement Learning](https://mlsanity.com/paper/2604.16918/) (2026)
- [Rollout-Level Advantage-Prioritized Experience Replay for GRPO](https://mlsanity.com/paper/2606.04560/) (2026)
- [IRDS: Interpretable RLVR Data Selection via Verifier-Coupled Sparse Autoencoder Coverage](https://mlsanity.com/paper/2605.28247/) (2026)
- [Evaluating Fuzz Testing for Reinforcement Learning Agents](https://mlsanity.com/paper/2607.24577/) (2026)

## Abstract (arXiv)

We introduce a problem-level prioritization framework for RL post-training of large language models. Building on insights from prioritized replay in deep RL, as well as prior observations that rollouts with intermediate success rates tend to produce stronger learning signals under methods such as GRPO, our approach selects problems according to a simple, model-driven priority score derived from empirical success statistics. In contrast to conventional curriculum strategies that emphasize easier tasks early in training, the resulting schedule naturally focuses training on problems that are neither consistently solved nor consistently failed, while deprioritizing those that contribute little gradient information. The method yields a continuously adapting and automatic prioritization process that requires no predefined difficulty tiers, auxiliary predictors, or external labels. We further introduce lightweight mechanisms for practical deployment, including heap-based prioritized sampling and periodic retesting of solved and unsolved problems to mitigate starvation and forgetting. Overall, the approach offers a principled and scalable alternative to manually designed curricula while aligning data selection directly with the dynamics of GRPO-based post-training.