mlsanityData to 14 Oct 2026

ThinkPrior: Zero-Rollout Difficulty Priors for Cold-Start Prompt Selection in RLVR

arXiv 2609.09075 ↗First posted 8 Sep 2026cs.LG cs.AIcited by 2

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

In reinforcement learning with verifiable rewards (RLVR) trained with group relative policy optimization (GRPO), the KL-free reward-advantage term studied here depends on within-group reward variation. If all rollouts in a group are correct or all are wrong, their group-relative advantages are identically zero; these zero-advantage silent groups provide no reward-advantage gradient, yet uniform sampling spends 39% of a run's rollouts on them. History-based prompt selection must first spend target-policy rollouts to estimate difficulty, creating a cold start with rollout waste; ThinkPrior instead uses an external anchor in one offline pass to construct a zero-rollout difficulty prior before the first target-policy rollout. The verifier-scored anchor pass rate supplies an external-anchor initialization for a Beta posterior; ThinkPrior selects by expected learnability and then updates from training outcomes, changing neither the loss nor the optimizer. On Qwen2.5-Math-7B across sixteen seeds, ThinkPrior more than halves early silent groups and cuts wasted rollouts through step 30 by nearly a fifth, while we detect no difference in final accuracy. On this 250-prompt pool the fixed-budget result is a reallocation rather than a net saving. The measured ThinkPrior+DAPO composition reduces generated rollouts by 10.6% while both arms retain the same 3840-rollout update budget. The prior requires no target-policy rollout before the first selection, but the posterior thereafter uses target-policy outcomes.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 24 of its 35 references that are in the map. First posted on arXiv 8 Sep 2026.

Builds on

24 of 35 references.

Closest ancestors · 9

Same direction, newest first

  1. 2026Small Generalizable Prompt Predictive Models Can Steer Efficient RL Post-Training of Large Reasoning Models2 Feb
  2. 2025Can Prompt Difficulty be Online Predicted for Accelerating RL Finetuning of Reasoning Models?7 Jul
  3. 2025Act Only When It Pays: Efficient Reinforcement Learning for LLM Reasoning via Selective Rollouts2 Jun
  4. 2025Online Difficulty Filtering for Reasoning Oriented Reinforcement Learning4 Apr
  5. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  6. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMs22 Jan
  7. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Training22 Nov
  8. 2023Let's Verify Step by Step31 May
  9. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 15

  1. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  3. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  4. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,392
  5. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,136
  6. 2015Prioritized Experience Replaycited by 1,099
  7. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,090
  8. 2023Data Selection for Language Models via Importance Resamplingcited by 329
  9. 2024tinyBenchmarks: evaluating LLMs with fewer examplescited by 310
  10. 2017Automated Curriculum Learning for Neural Networkscited by 257
  11. 2022Prioritized Training on Points that are Learnable, Worth Learning, and Not Yet Learntcited by 236
  12. 2024Superfiltering: Weak-to-Strong Data Filtering for Fast Instruction-Tuningcited by 138
3 more
  1. 2025Prompt Curriculum Learning for Efficient LLM Post-Trainingcited by 45
  2. 2019Learning Latent Parameters without Human Response Patterns: Item Response Theory with Artificial Crowdscited by 40
  3. 2026sGPO: Trading Inference FLOPs for Training Efficiency in RLVRcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Prompts Live on an Arc: Gaussian Curricula in Fisher--Rao Coordinates for Rollout-Efficient GRPOcited by 0
  2. 2026Tev: Scaling Long-Horizon Agentic Environments with Dense Progress Rewardscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Where to Spend Rollouts: Hit-Utility Optimal Rollout Allocation for Group-Based RLVR
  2. 2026Early Verdicts, Better Budgets: Sequential Adaptive Rollout Allocation for Compute-Efficient RLVR
  3. 2026Hindsight-Privileged RLVR: Single-Rollout Token Credit Without Group Sampling
  4. 2026Contextual Rollout Bandits for Reinforcement Learning with Verifiable Rewards
  5. 2025Improving Sampling Efficiency in RLVR through Adaptive Rollout and Response Reuse
  6. 2026Your Group-Relative Advantage Is Biased
  7. 2026Selective Rollout: Mid-Trajectory Termination for Multi-Sample Agent RL
  8. 2026SALT: When More Rollouts Don't Help in Group-Based Policy Optimization and How to Make Them Matter