mlsanityData to 14 Oct 2026

Can Prompt Difficulty be Online Predicted for Accelerating RL Finetuning of Reasoning Models?

arXiv 2507.04632 ↗First posted 7 Jul 2025cs.AI cs.LGcited by 52

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Recent advances have witnessed the effectiveness of reinforcement learning (RL) finetuning in enhancing the reasoning capabilities of large language models (LLMs). The optimization process often requires numerous iterations to achieve satisfactory performance, resulting in high computational costs due to the need for frequent prompt evaluations under intensive LLM interactions and repeated policy updates. Appropriate online prompt selection methods reduce iteration steps by prioritizing informative prompts during training, while the pipeline's reliance on exhaustive prompt evaluation and subset selection for optimization still incurs substantial computational overhead due to frequent LLM inference calls. Distinguished from these direct evaluate-then-select schemes, this work investigates iterative approximate evaluation for arbitrary prompts and introduces Model Predictive Prompt Selection (MoPPS), a Bayesian risk-predictive framework that online estimates prompt difficulty without requiring costly LLM interactions. Technically, MoPPS models each prompt's success rate as a latent variable, performs streaming Bayesian inference, and employs posterior sampling in a constructed multi-armed bandit machine, enabling sample efficient and adaptive prompt selection. Extensive experiments across mathematics, planning, and vision-based geometry tasks show that MoPPS reliably predicts prompt difficulty and accelerates training with significantly reduced LLM rollouts. Our code is available at https://github.com/thu-rllab/MoPPS.

Key facts. Cited by 52 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 45 of its 65 references that are in the map. First posted on arXiv 7 Jul 2025.

Builds on

45 of 65 references.

Closest ancestors · 23

Same direction, newest first

  1. 2025Act Only When It Pays: Efficient Reinforcement Learning for LLM Reasoning via Selective Rollouts2 Jun
  2. 2025ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Models30 May
  3. 2025Self-Evolving Curriculum for LLM Reasoning20 May
  4. 2025Reinforcement Learning for Reasoning in Large Language Models with One Training Example29 Apr
  5. 2025SRPO: A Cross-Domain Implementation of Large-Scale Reinforcement Learning on LLM19 Apr
  6. 2025A Minimalist Approach to LLM Reasoning: from Rejection Sampling to Reinforce15 Apr
  7. 2025VAPO: Efficient and Reliable Reinforcement Learning for Advanced Reasoning Tasks7 Apr
  8. 2025Online Difficulty Filtering for Reasoning Oriented Reinforcement Learning4 Apr
  9. 2025Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Model31 Mar
  10. 2025CPPO: Accelerating the Training of Group Relative Policy Optimization-Based Reasoning Models28 Mar
  11. 2025Understanding R1-Zero-Like Training: A Critical Perspective26 Mar
  12. 2025SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wild24 Mar
11 more
  1. 2025FastCuRL: Curriculum Reinforcement Learning with Stage-wise Context Scaling for Efficient Training R1-like Reasoning Models21 Mar
  2. 2025Reinforcement Learning for Reasoning in Small LLMs: What Works and What Doesn't20 Mar
  3. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  4. 2025What's Behind PPO's Collapse in Long-CoT? Value Optimization Holds the Secret3 Mar
  5. 2025LIMR: Less is More for RL Scaling17 Feb
  6. 2025Process Reinforcement through Implicit Rewards3 Feb
  7. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning22 Jan
  8. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMs22 Jan
  9. 2025REINFORCE++: Stabilizing Critic-Free Policy Optimization with Global Advantage Normalization4 Jan
  10. 2024RLHF Workflow: From Reward Modeling to Online RLHF13 May
  11. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 22

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  4. 2025Qwen2.5-VL Technical Reportcited by 5,587
  5. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  6. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
  7. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
  8. 2024OpenAI o1 System Cardcited by 1,244
  9. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,136
  10. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,090
  11. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 712
  12. 2025SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-trainingcited by 672
10 more
  1. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 656
  2. 2025LIMO: Less is More for Reasoningcited by 498
  3. 2024LLaVA-CoT: Let Vision Language Models Reason Step-by-Stepcited by 454
  4. 2017Variational Continual Learningcited by 339
  5. 2023Secrets of RLHF in Large Language Models Part I: PPOcited by 229
  6. 2024VinePPO: Refining Credit Assignment in RL Training of LLMscited by 183
  7. 2025Light-R1: Curriculum SFT, DPO and RL for Long COT from Scratch and Beyondcited by 175
  8. 2025Concise Reasoning via Reinforcement Learningcited by 79
  9. 2025Model Predictive Task Sampling for Efficient and Robust Adaptationcited by 21
  10. 2025Fast and Robust: Task Sampling with Posterior and Diversity Synergies for Adaptive Decision-Makers in Randomized Environmentscited by 12

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 16; 2026: 36.

52 papers.

Sort:

By area:

  1. 2025No Prompt Left Behind: Exploiting Zero-Variance Prompts in LLM Reinforcement Learning via Entropy-Guided Advantage Shapingcited by 50
  2. 2025Prompt Curriculum Learning for Efficient LLM Post-Trainingcited by 45
  3. 2025RollPacker: Mitigating Long-Tail Rollouts for Fast, Synchronous RL Post-Trainingcited by 35
  4. 2025FlyLoRA: Boosting Task Decoupling and Parameter Efficiency via Implicit Rank-Wise Mixture-of-Expertscited by 35
  5. 2025Single-stream Policy Optimizationcited by 28
  6. 2025Diversity-Incentivized Exploration for Versatile Reasoningcited by 28
  7. 2025Reinforce-Ada: An Adaptive Sampling Framework under Non-linear RL Objectivescited by 25
  8. 2025CurES: From Gradient Analysis to Efficient Curriculum Learning for Reasoning LLMscited by 19
  9. 2025XRPO: Pushing the limits of GRPO with Targeted Exploration and Exploitationcited by 18
  10. 2026Prefix Teach, Suffix Fade: Local Teachability Collapse in Strong-to-Weak On-Policy Distillationcited by 16
  11. 2025Utility-Diversity Aware Online Batch Selection for LLM Supervised Fine-tuningcited by 15
  12. 2025BOTS: A Unified Framework for Bayesian Online Task Selection in LLM Reinforcement Finetuningcited by 14
38 more
  1. 2026Small Generalizable Prompt Predictive Models Can Steer Efficient RL Post-Training of Large Reasoning Modelscited by 14
  2. 2025Fly-CL: A Fly-Inspired Framework for Enhancing Efficient Decorrelation and Reduced Training Time in Pre-trained Model-based Continual Representation Learningcited by 11
  3. 2026Dynamics-Predictive Sampling for Active RL Finetuning of Large Reasoning Modelscited by 11
  4. 2026TRACE: A Unified Rollout Budget Allocation Framework for Efficient Agentic Reinforcement Learningcited by 10
  5. 2026RLVR without Ineffective Samples: Group Prioritized Off-Policy Optimization for LLM Reasoningcited by 9
  6. 2026Buffer Matters: Unleashing the Power of Off-Policy Reinforcement Learning in Large Language Model Reasoningcited by 8
  7. 2026Prompt replay: speeding up grpo with on-policy reuse of high-signal promptscited by 8
  8. 2025BTCChat: Advancing Remote Sensing Bi-temporal Change Captioning with Multimodal Large Language Modelcited by 7
  9. 2025Efficient Reinforcement Learning for Large Language Models with Intrinsic Explorationcited by 7
  10. 2026Composition-RL: Compose Your Verifiable Prompts for Reinforcement Learning of Large Language Modelscited by 6
  11. 2026Efficient RLVR Training via Weighted Mutual Information Data Selectioncited by 6
  12. 2025Kuramoto Orientation Diffusion Modelscited by 5
  13. 2026Where to Spend Rollouts: Hit-Utility Optimal Rollout Allocation for Group-Based RLVRcited by 5
  14. 2026Spend Your Rollouts Where It Counts: Rollout Allocation for Group-Based RL Post-Trainingcited by 5
  15. 2026HeaPA: Difficulty-Aware Heap Sampling and On-Policy Query Augmentation for LLM Reinforcement Learningcited by 4
  16. 2026Goldilocks RL: Tuning Task Difficulty to Escape Sparse Rewards for Reasoningcited by 4
  17. 2025DTop-p MoE: Sparsity-Controlled Dynamic Top-p MoE for Foundation Model Pre-trainingcited by 3
  18. 2026Limited Reasoning Space: The cage of long-horizon reasoning in LLMscited by 3
  19. 2026Supervised Fine-Tuning versus Reinforcement Learning: A Study of Post-Training Methods for Large Language Modelscited by 3
  20. 2026Listwise Policy Optimization: Group-based RLVR as Target-Projection on the LLM Response Simplexcited by 3
  21. 2026DARE: Difficulty-Adaptive Reinforcement Learning with Co-Evolved Difficulty Estimationcited by 2
  22. 2026Kalman Meets Curriculum: Efficient Dynamic Prompt Selection for Adaptive RL Finetuningcited by 2
  23. 2026LEEPS: Latent-Guided Explore-Exploit Prompt Sampling for Efficient RLVR in Large Language Modelscited by 2
  24. 2026Beyond Solvability: Task Learnability as a Static Prior for LLM RL Post-Trainingcited by 2
  25. 2026ThinkPrior: Zero-Rollout Difficulty Priors for Cold-Start Prompt Selection in RLVRcited by 2
  26. 2026Beyond Normalization: Rethinking the Partition Function as a Difficulty Scheduler for RLVRcited by 1
  27. 2026Learning to Solve Hard Problems in RL for LLMs by Never Giving Upcited by 1
  28. 2026Balancing the Reasoning Load: Difficulty-Differentiated Policy Optimization with Length Redistribution for Efficient and Robust Reinforcement Learningcited by 0
  29. 2026A Survey of Reinforcement Learning for Large Language Models under Data Scarcity: Challenges and Solutionscited by 0
  30. 2026Manifold Bandits: Bayesian Curriculum Learning over the Latent Geometry of Large Language Modelscited by 0
  31. 2026CVPO: Enhancing LLM Reinforcement Learning Reasoning via Value-Variance Adaptation and Dynamic Curriculum Learningcited by 0
  32. 2026PERO: Efficient Robust Post-Training Foundation Models for Encrypted Traffic Classificationcited by 0
  33. 2026PAC: Progress-Augmented Advantage Curriculum for Multi-Task Reinforcement Learning of LLMscited by 0
  34. 2026Multi-Agent System Search via Active Substructure-aware Policy Optimizationcited by 0
  35. 2026You Only Edit Once: Incentivizing In-Context Capability of LLMs via Local Demonstration Refinementcited by 0
  36. 2026Selecting Diverse SFT Traces Improves Post-RL Generalizationcited by 0
  37. 2026SERA: Scale-Equalized Rollout Allocation for Maximum Likelihood Reinforcement Learningcited by 0
  38. 2026A Step Harder, Not a Step Too Far: The Benefits and Limits of Curriculum Difficulty Progression in GRPOcited by 0

50 most cited of 52 All 52 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Accelerating RL for LLM Reasoning with Optimal Advantage Regression
  2. 2026Train at Moving Edge: Online-Verified Prompt Selection for Efficient RL Training of Large Reasoning Model
  3. 2025SPEED-RL: Faster Training of Reasoning Models via Online Curriculum Learning
  4. 2025Prompt-R1: Collaborative Automatic Prompting Framework via End-to-end Reinforcement Learning
  5. 2025Reinforcement Learning Meets Large Language Models: A Survey of Advancements and Applications Across the LLM Lifecycle
  6. 2026MaPP: A Unified Marginalized Posterior-Predictive Framework for Data-Efficient RLVR
  7. 2025Do Not Let Low-Probability Tokens Over-Dominate in RL for LLMs
  8. 2026Rethinking RL for LLM Reasoning: It's Sparse Policy Selection, Not Capability Learning