mlsanityData to 14 Oct 2026

Where Rollouts Begin: Low-Load, High-Leverage First-Token Diversification for RLVR

arXiv 2605.28295 ↗First posted 27 May 2026cs.AI cs.CL cs.LGcited by 2

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Reinforcement learning with verifiable rewards (RLVR) trains reasoning models without labeled trajectories, using groups of verifier-scored rollouts to explore alternative reasoning paths. Limited rollout diversity is a central bottleneck, typically addressed through adjustments to temperature, prefixes, or rollout selection. We identify the first token of the response as a structurally distinct target for diversification, largely overlooked in prior work. We find that the first-token distribution is sharply concentrated and only weakly related to downstream correctness, as lower-probability candidates can yield similarly accurate responses. Diversifying the first token can therefore broaden the reasoning paths explored within each rollout group with little loss in response quality. Motivated by this observation, we introduce REFT (Rollout Exploration with First-Token Diversification), a lightweight modification to RLVR. REFT samples first tokens uniformly from the policy's top-N candidates and allocates rollouts evenly across the sampled tokens, leaving the rest of the pipeline unchanged. We evaluate REFT on eight models spanning multiple architectures and sizes (0.5B-14B), with mathematical reasoning and code-generation tasks under GRPO and DAPO. Across these settings, REFT consistently improves Pass@1, Pass@8, and Pass@64. It also outperforms competing diversification methods at every evaluated budget, incurring the lowest rollout cost.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 52 of its 61 references that are in the map. First posted on arXiv 27 May 2026.

Builds on

52 of 61 references.

Closest ancestors · 29

Same direction, newest first

  1. 2026ERPO: Token-Level Entropy-Regulated Policy Optimization for Large Reasoning Models30 Mar
  2. 2026DSDR: Dual-Scale Diversity Regularization for Exploration in LLM Reasoning23 Feb
  3. 2026How to Allocate, How to Learn? Dynamic Rollout Allocation and Advantage Modulation for Policy Optimization22 Feb
  4. 2026Learning to Explore with Parameter-Space Noise: A Deep Dive into Parameter-Space Noise for Reinforcement Learning with Verifiable Rewards30 Jan
  5. 2026Training Reasoning Models on Saturated Problems via Failure-Prefix Conditioning28 Jan
  6. 2026Reuse your FLOPs: Scaling RL on Hard Problems by Conditioning on Very Off-Policy Prefixes26 Jan
  7. 2026Reinforced Efficient Reasoning via Semantically Diverse Exploration8 Jan
  8. 2025Lookahead Tree-Based Rollouts for Enhanced Trajectory-Level Exploration in Reinforcement Learning with Verifiable Rewards28 Oct
  9. 2025Exploring Multi-Temperature Strategies for Token- and Rollout-Level Control in RLVR10 Oct
  10. 2025Let it Calm: Exploratory Annealed Decoding for Verifiable Reinforcement Learning6 Oct
  11. 2025Diversity-Incentivized Exploration for Versatile Reasoning30 Sep
  12. 2025Improving Sampling Efficiency in RLVR through Adaptive Rollout and Response Reuse30 Sep
17 more
  1. 2025No Prompt Left Behind: Exploiting Zero-Variance Prompts in LLM Reinforcement Learning via Entropy-Guided Advantage Shaping26 Sep
  2. 2025Outcome-based Exploration for LLM Reasoning8 Sep
  3. 2025Staying in the Sweet Spot: Responsive Reasoning Evolution via Capability-Adaptive Hint Scaffolding8 Sep
  4. 2025Pass@k Training for Adaptively Balancing Exploration and Exploitation of Large Reasoning Models14 Aug
  5. 2025First Return, Entropy-Eliciting Explore9 Jul
  6. 2025Act Only When It Pays: Efficient Reinforcement Learning for LLM Reasoning via Selective Rollouts2 Jun
  7. 2025Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning2 Jun
  8. 2025The Surprising Effectiveness of Negative Reinforcement in LLM Reasoning2 Jun
  9. 2025ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Models30 May
  10. 2025The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models28 May
  11. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?18 Apr
  12. 2025Not All Rollouts are Useful: Down-Sampling Rollouts in LLM Reinforcement Learning18 Apr
  13. 2025Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Model31 Mar
  14. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  15. 2025Big-Math: A Large-Scale, High-Quality Math Dataset for Reinforcement Learning in Language Models24 Feb
  16. 2023Let's Verify Step by Step31 May
  17. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 23

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  4. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  5. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  6. 2024Qwen2.5 Technical Reportcited by 4,942
  7. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 2,004
  8. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,848
  9. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,807
  10. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,391
  11. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,301
  12. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,228
11 more
  1. 2024Qwen2.5-Coder Technical Reportcited by 1,203
  2. 2020MPNet: Masked and Permuted Pre-training for Language Understandingcited by 981
  3. 2023Understanding the Effects of RLHF on LLM Generalisation and Diversitycited by 396
  4. 2025The Invisible Leash: Why RLVR May or May Not Escape Its Origincited by 97
  5. 2025TreeRL: LLM Reinforcement Learning with On-Policy Tree Searchcited by 80
  6. 2025QeRL: Beyond Efficiency -- Quantization-enhanced Reinforcement Learning for LLMscited by 16
  7. 2026Where does output diversity collapse in post-training?cited by 11
  8. 2026QaRL: Rollout-Aligned Quantization-Aware RL for Fast and Stable Training under Training--Inference Mismatchcited by 7
  9. 2026SRT: Accelerating Reinforcement Learning via Speculative Rollout with Tree-Structured Cachecited by 5
  10. 2026Jackpot: Optimal Budgeted Rejection Sampling for Extreme Actor-Policy Mismatch Reinforcement Learningcited by 5
  11. 2026Entropy-Tree: Tree-Based Decoding with Entropy-Guided Explorationcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Locked at the Entrance, Open Inside: Where RLVR Narrows the Solution Spacecited by 0
  2. 2026Video-HopChain: Multi-Hop Questions and Confidence-Gated Exploration for Video Reasoning Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026PAIR: Pairwise-Aware Inclusion Reweighting for Adaptive Rollout Allocation in RLVR
  2. 2026RolloutPipe: Overlapping Pipelined Rollout and Training in Disaggregated On-Policy LLM Reinforcement Learning
  3. 2026Hindsight-Privileged RLVR: Single-Rollout Token Credit Without Group Sampling
  4. 2026Early Verdicts, Better Budgets: Sequential Adaptive Rollout Allocation for Compute-Efficient RLVR
  5. 2026Learning as Reasoning Unfolds: Progressive Rollout Allocation for Efficient Reinforcement Learning
  6. 2026Contextual Rollout Bandits for Reinforcement Learning with Verifiable Rewards
  7. 2026The Model Knows Another Way: Strategy Switching for Effective RLVR Exploration
  8. 2025SPEC-RL: Accelerating On-Policy Reinforcement Learning with Speculative Rollouts