mlsanityData to 7 Oct 2026

Prune as You Generate: Online Rollout Pruning for Faster and Better RLVR

arXiv 2603.24840 ↗First posted 25 Mar 2026cs.CLcited by 8

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Reinforcement Learning with Verifiable Rewards (RLVR) has significantly advanced the reasoning capabilities of Large Language Models (LLMs). However, methods such as GRPO and DAPO suffer from substantial computational cost, since they rely on sampling many rollouts for each prompt. Moreover, in RLVR the relative advantage is often sparse: many samples become nearly all-correct or all-incorrect, yielding low within-group reward variance and thus weak learning signals. In this paper, we introduce arrol (Accelerating RLVR via online Rollout Pruning), an online rollout pruning method that prunes rollouts during generation while explicitly steering the surviving ones more correctness-balanced to enhance learning signals. Specifically, arrol trains a lightweight quality head on-the-fly to predict the success probability of partial rollouts and uses it to make early pruning decisions. The learned quality head can further weigh candidates to improve inference accuracy during test-time scaling. To improve efficiency, we present a system design that prunes rollouts inside the inference engine and re-batches the remaining ones for log-probability computation and policy updates. Across GRPO and DAPO on Qwen-3 and LLaMA-3.2 models (1B-8B), arrol improves average accuracy by +2.30 to +2.99 while achieving up to 1.7x training speedup, and yielding up to +8.33 additional gains in average accuracy in test-time scaling. The code is available at https://github.com/Hsu1023/ARRoL.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 51 of its 95 references that are in the map. First posted on arXiv 25 Mar 2026.

Builds on

51 of 95 references.

Closest ancestors · 8

Same direction, newest first

  1. 2025History Rhymes: Accelerating LLM Reinforcement Learning with RhymeRL26 Aug
  2. 2025Act Only When It Pays: Efficient Reinforcement Learning for LLM Reasoning via Selective Rollouts2 Jun
  3. 2025S-GRPO: Early Exit via Reinforcement Learning in Reasoning Models12 May
  4. 2025Not All Rollouts are Useful: Down-Sampling Rollouts in LLM Reinforcement Learning18 Apr
  5. 2025Online Difficulty Filtering for Reasoning Oriented Reinforcement Learning4 Apr
  6. 2025CPPO: Accelerating the Training of Group Relative Policy Optimization-Based Reasoning Models28 Mar
  7. 2025Understanding R1-Zero-Like Training: A Critical Perspective26 Mar
  8. 2023Let's Verify Step by Step31 May

Other references · 43

  1. 2017Attention Is All You Needcited by 59,932
  2. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  3. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  4. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  5. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,928
  6. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  7. 2020Knowledge Distillation: A Surveycited by 1,578
  8. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,359
  9. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,099
  10. 2025Scalable Best-of-N Selection for Large Language Models via Self-Certaintycited by 182
  11. 2024DuQuant: Distributing Outliers via Dual Transformation Makes Stronger Quantized LLMscited by 165
  12. 2024Discrete-state Continuous-time Diffusion for Graph Generationcited by 73
31 more
  1. 2025SD-LoRA: Scalable Decoupled Low-Rank Adaptation for Class Incremental Learningcited by 60
  2. 2026Mem-Gallery: Benchmarking Multimodal Long-Term Conversational Memory for MLLM Agentscited by 49
  3. 2024How Efficient is LLM-Generated Code? A Rigorous & High-Standard Benchmarkcited by 39
  4. 2024MoPE-CLIP: Structured Pruning for Efficient Vision-Language Models with Module-wise Pruning Error Metriccited by 30
  5. 2023Ensuring User-side Fairness in Dynamic Recommender Systemscited by 27
  6. 2026QuantVLA: Scale-Calibrated Post-Training Quantization for Vision-Language-Action Modelscited by 25
  7. 2023Hierarchical Multi-Marginal Optimal Transport for Network Alignmentcited by 24
  8. 2025Quantization Meets dLLMs: A Systematic Study of Post-training Quantization for Diffusion LLMscited by 21
  9. 2024Logic Query of Thoughts: Guiding Large Language Models to Answer Complex Logic Queries with Knowledge Graphscited by 20
  10. 2025Pave Your Own Path: Graph Gradual Domain Adaptation on Fused Gromov-Wasserstein Geodesicscited by 17
  11. 2025TUCKET: A Tensor Time Series Data Structure for Efficient and Accurate Factor Analysis over Time Rangescited by 12
  12. 2025ResMoE: Space-efficient Compression of Mixture of Experts LLMs via Residual Restorationcited by 12
  13. 2024Ask, and it shall be given: On the Turing completeness of promptingcited by 10
  14. 2024DOGR: Towards Versatile Visual Document Grounding and Referringcited by 10
  15. 2025EfficientLLM: Scalable Pruning-Aware Pretraining for Architecture-Agnostic Edge Language Modelscited by 10
  16. 2025Joint Optimal Transport and Embedding for Network Alignmentcited by 10
  17. 2025PLANETALIGN: A Comprehensive Python Library for Benchmarking Network Alignmentcited by 10
  18. 2025LRQ-DiT: Log-Rotation Post-Training Quantization of Diffusion Transformers for Image and Video Generationcited by 10
  19. 2025Hierarchical LoRA MoE for Efficient CTR Model Scalingcited by 10
  20. 2025Harnessing Consistency for Robust Test-Time LLM Ensemblecited by 10
  21. 2025CoFiRec: Coarse-to-Fine Tokenization for Generative Recommendationcited by 10
  22. 2025Latte: Collaborative Test-Time Adaptation of Vision-Language Models in Federated Learningcited by 9
  23. 2026Influence-Preserving Proxies for Gradient-Based Data Selection in LLM Fine-tuningcited by 9
  24. 2023Neural Exploitation and Exploration of Contextual Banditscited by 7
  25. 2025Scale Up Composed Image Retrieval Learning via Modification Text Generationcited by 7
  26. 2025CGES: Confidence-Guided Early Stopping for Efficient and Accurate Self-Consistencycited by 6
  27. 2025PowerGrow: Feasible Co-Growth of Structures and Dynamics for Power Grid Synthesiscited by 5
  28. 2025MedREK: Retrieval-Based Editing for Medical LLMs with Key-Aware Promptscited by 5
  29. 2025Enhancing Test-Time Scaling of Large Language Models with Hierarchical Retrieval-Augmented MCTScited by 4
  30. 2026AdaFuse: Adaptive Ensemble Decoding with Test-Time Scaling for LLMscited by 4
  31. 2026Graph homophily booster: Reimagining the role of discrete features in heterophilic graph learningcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 8.

8 papers.

  1. 2026TRACE: A Unified Rollout Budget Allocation Framework for Efficient Agentic Reinforcement Learningcited by 9
  2. 2026DuQuant++: Fine-grained Rotation Enhances Microscaling FP4 Quantizationcited by 8
  3. 2026DUET: Optimize Token-Budget Allocation for Reinforcement Learning with Verifiable Rewardscited by 3
  4. 2026Predict, Don't Iterate: Efficient Adaptive-Length Infilling for Diffusion Language Modelscited by 3
  5. 2026IV-CoT: Implicit Visual Chain-of-Thought for Structure-Aware Text-to-Image Generationcited by 2
  6. 2026Benchmarking Trustworthiness of SLMs: Pre-trained vs. Compressedcited by 2
  7. 2026Extending Confidence-Based Text2Cypher with Grammar and Schema Aware Filteringcited by 1
  8. 2026Improving the Diversity of LLM Outputs without a Trade-offcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026RolloutPipe: Overlapping Pipelined Rollout and Training in Disaggregated On-Policy LLM Reinforcement Learning
  2. 2026Contextual Rollout Bandits for Reinforcement Learning with Verifiable Rewards
  3. 2025RollPacker: Mitigating Long-Tail Rollouts for Fast, Synchronous RL Post-Training
  4. 2026Beyond Negative Rollouts: Positive-Only Policy Optimization with Implicit Negative Gradients
  5. 2026Generate, Filter, Control, Replay: A Comprehensive Survey of Rollout Strategies for LLM Reinforcement Learning
  6. 2025Depth-Breadth Synergy in RLVR: Unlocking LLM Reasoning Gains with Adaptive Exploration
  7. 2026Learning as Reasoning Unfolds: Progressive Rollout Allocation for Efficient Reinforcement Learning
  8. 2025SPEC-RL: Accelerating On-Policy Reinforcement Learning with Speculative Rollouts