mlsanityData to 7 Oct 2026

RollVerify: Bridging Efficiency and Accuracy in Long-Tail Rollout Reinforcement Learning

arXiv 2610.09914 ↗First posted 7 Oct 2026cs.LG cs.AIcited by 0

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Reinforcement learning is crucial for improving large language models' reasoning and generalization. It relies on massive rollouts whose lengths become increasingly long-tailed as context windows grow. In on-policy training, these long-tail rollouts can result in GPU bubbles, reducing system utilization and limiting RL scalability. Asynchronous or partial-rollout methods improve throughput by relaxing synchronization, but inevitably introduce stale off-policy samples (trajectories) that may hurt final accuracy. Existing approaches mainly mitigate this off-policy issue by reweighting off-policy samples during training, yet they can still leave a performance gap compared to fully on-policy training. In this work, rather than passively reweighting samples during training, we propose RollVerify, a lightweight RL framework built on partial rollout that actively verifies and repairs samples before they enter training. Specifically, it introduces an off-policy shift metric OPS, to quantify the off-policy deviation of partially generated trajectories. Guided by the OPS constraint, RollVerify performs both sequence-level and token-level verification to identify and truncate invalid suffixes of trajectories. This yields high-quality samples that protect the models' accuracy while preserving the efficiency gains of partial rollout. Experiments on mathematical and tool-assisted mathematical reasoning show that RollVerify achieves accuracy comparable to on-policy training while reducing training cost. Additional code-generation results provide preliminary evidence beyond mathematics.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 28 of its 39 references that are in the map. First posted on arXiv 7 Oct 2026.

Builds on

28 of 39 references.

Closest ancestors · 9

Same direction, newest first

  1. 2025Soft Adaptive Policy Optimization25 Nov
  2. 2025RollPacker: Mitigating Long-Tail Rollouts for Fast, Synchronous RL Post-Training25 Sep
  3. 2025Reinforcement Learning Optimization for Large-Scale Learning: An Efficient and User-Friendly Scaling Library6 Jun
  4. 2025AReaL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoning30 May
  5. 2025MiMo: Unlocking the Reasoning Potential of Language Model -- From Pretraining to Posttraining12 May
  6. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  7. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning22 Jan
  8. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMs22 Jan
  9. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 19

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  3. 2025Qwen3 Technical Reportcited by 8,195
  4. 2024Qwen2.5 Technical Reportcited by 4,713
  5. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 1,980
  6. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 1,906
  7. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
  8. 2025Group Sequence Policy Optimizationcited by 788
  9. 2024DeepSeek-V3 Technical Reportcited by 724
  10. 2023PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallelcited by 641
  11. 2026GLM-5: from Vibe Coding to Agentic Engineeringcited by 502
  12. 2026Kimi K2.5: Visual Agentic Intelligencecited by 449
7 more
  1. 2025ReTool: Reinforcement Learning for Strategic Tool Use in LLMscited by 377
  2. 2025MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attentioncited by 248
  3. 2025Stabilizing MoE Reinforcement Learning by Aligning Training and Inference Routerscited by 67
  4. 2025StreamRL: Scalable, Heterogeneous, and Elastic RL for LLMs with Disaggregated Stream Generationcited by 56
  5. 2024Optimizing RLHF Training for Large Language Models with Stage Fusioncited by 41
  6. 2025AsyncFlow: An Asynchronous Streaming RL Framework for Efficient LLM Post-Trainingcited by 37
  7. 2026VESPO: Variational Sequence-Level Soft Policy Optimization for Stable Off-Policy LLM Trainingcited by 9

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025APRIL: Active Partial Rollouts in Reinforcement Learning to Tame Long-tail Generation
  2. 2025Prosperity before Collapse: How Far Can Off-Policy RL Reach with Stale Data on LLMs?
  3. 2025SPEC-RL: Accelerating On-Policy Reinforcement Learning with Speculative Rollouts
  4. 2026RolloutPipe: Overlapping Pipelined Rollout and Training in Disaggregated On-Policy LLM Reinforcement Learning
  5. 2026Reusing Rollouts under Policy Lag: Prefix-Normalized Policy Optimization for LLM Reinforcement Learning
  6. 2025CoPRIS: Efficient and Stable Reinforcement Learning via Concurrency-Controlled Partial Rollout with Importance Sampling
  7. 2026Generate, Filter, Control, Replay: A Comprehensive Survey of Rollout Strategies for LLM Reinforcement Learning
  8. 2026Single-Rollout Asynchronous Optimization for Agentic Reinforcement Learning