mlsanityData to 7 Oct 2026

QuRL: Efficient Reinforcement Learning with Quantized Rollout

arXiv 2602.13953 ↗First posted 15 Feb 2026cs.LGcited by 6

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Reinforcement learning with verifiable rewards (RLVR) has become a trending paradigm for training reasoning large language models (LLMs). However, due to the autoregressive decoding nature of LLMs, the rollout process becomes the efficiency bottleneck of RL training, consisting of up to 70% of the total training time. In this work, we propose Quantized Reinforcement Learning (QuRL) that uses a quantized actor for accelerating the rollout. We address two challenges in QuRL. First, we propose Adaptive Clipping Range (ACR) that dynamically adjusts the clipping ratio based on the policy ratio between the full-precision actor and the quantized actor, which is essential for mitigating long-term training collapse. Second, we identify the weight update problem, where weight changes between RL steps are extremely small, making it difficult for the quantization operation to capture them effectively. We mitigate this problem through the invariant scaling technique that reduces quantization noise and increases weight update. We evaluate our method with INT8 and FP8 quantization experiments on DeepScaleR and DAPO, and achieve 20% to 80% faster rollout during training.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 41 of its 54 references that are in the map. First posted on arXiv 15 Feb 2026.

Builds on

41 of 54 references.

Closest ancestors · 8

Same direction, newest first

  1. 2025History Rhymes: Accelerating LLM Reinforcement Learning with RhymeRL26 Aug
  2. 2025Act Only When It Pays: Efficient Reinforcement Learning for LLM Reasoning via Selective Rollouts2 Jun
  3. 2025AReaL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoning30 May
  4. 2025Not All Rollouts are Useful: Down-Sampling Rollouts in LLM Reinforcement Learning18 Apr
  5. 2025VAPO: Efficient and Reliable Reinforcement Learning for Advanced Reasoning Tasks7 Apr
  6. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  7. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMs22 Jan
  8. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 33

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  5. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  6. 2024Qwen2.5 Technical Reportcited by 4,713
  7. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  8. 2015Trust Region Policy Optimizationcited by 2,503
  9. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 1,980
  10. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  11. 2016Concrete Problems in AI Safetycited by 1,582
  12. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,359
21 more
  1. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
  2. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,107
  3. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,099
  4. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,090
  5. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 881
  6. 2021A Survey of Quantization Methods for Efficient Neural Network Inferencecited by 798
  7. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 604
  8. 2021A White Paper on Neural Network Quantizationcited by 453
  9. 2023OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Modelscited by 403
  10. 2023QuIP: 2-Bit Quantization of Large Language Models With Guaranteescited by 397
  11. 2023LLM-QAT: Data-Free Quantization Aware Training for Large Language Modelscited by 320
  12. 2024Omni-MATH: A Universal Olympiad Level Mathematic Benchmark For Large Language Modelscited by 277
  13. 2024Extreme Compression of Large Language Models via Additive Quantizationcited by 223
  14. 2022Outlier Suppression: Pushing the Limit of Low-bit Transformer Language Modelscited by 176
  15. 2024Imitate, Explore, and Self-Improve: A Reproduction Report on Slow-thinking Reasoning Systemscited by 162
  16. 2024Language Models Learn to Mislead Humans via RLHFcited by 118
  17. 2020Mirror Descent Policy Optimizationcited by 102
  18. 2024GPTVQ: The Blessing of Dimensionality for LLM Quantizationcited by 78
  19. 2023QLLM: Accurate and Efficient Low-Bitwidth Quantization for Large Language Modelscited by 63
  20. 2021Batch size-invariance for policy optimizationcited by 33
  21. 2025LO-BCQ: Block Clustered Quantization for 4-bit (W4A4) LLM Inferencecited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 6.

6 papers.

  1. 2026Accelerating RL Post-Training Rollouts via System-Integrated Speculative Decodingcited by 8
  2. 2026EfficientRollout: System-Aware Self-Speculative Decoding for RL Rolloutscited by 5
  3. 2026FP4 Explore, BF16 Train: Diffusion Reinforcement Learning via Efficient Rollout Scalingcited by 2
  4. 2026QUADS: Stabilizing NVFP4 Reinforcement Learning for MoE via QUantization-error Alignment across Dual Sidescited by 1
  5. 2026Reformulate LLM Reinforcement Learning for Efficient Training under Black-box Discrepancycited by 0
  6. 2026Towards Full Pipeline FP8 Reinforcement Learning for LLMscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026QaRL: Rollout-Aligned Quantization-Aware RL for Fast and Stable Training under Training--Inference Mismatch
  2. 2025QeRL: Beyond Efficiency -- Quantization-enhanced Reinforcement Learning for LLMs
  3. 2026TRACE: Rollout-Guided Quantization-Aware Training for FP4 Reinforcement Learning of MoE Language Models
  4. 2026FP8-RL: A Practical and Stable Low-Precision Stack for LLM Reinforcement Learning
  5. 2026Jet-RL: Enabling On-Policy FP8 Reinforcement Learning with Unified Training and Rollout Precision Flow
  6. 2025BroRL: Scaling Reinforcement Learning via Broadened Exploration
  7. 2026AIS: Adaptive Importance Sampling for Quantized RL
  8. 2026What are Key Factors for Updates in RL for LLM Reasoning?