mlsanityData to 14 Oct 2026

KV4-RL: Unlocking Faster Asynchronous Rollouts for RLVR with INT4 KV Cache Quantization

arXiv 2610.14448 ↗First posted 10 Oct 2026cs.LG cs.AIcited by 0

On the map: Efficient LLM inference and serving → KV cache quantization also fits: KV cache optimization

Abstract (arXiv)

Asynchronous rollout improves the efficiency of reinforcement learning with verifiable rewards (RLVR) by reducing under-utilization during long-tail generation. Sustained concurrency also concentrates context processing into fewer decode steps, increasing the KV workload per step. We characterize this KV concentration and find that KV cache quantization delivers substantially greater acceleration than weight-only and weight-activation quantization in our matched-workload comparisons. However, INT4 KV rollout can degrade learning even with token-level truncated importance sampling (TIS). We propose KV4-RL, which combines calibration-free INT4 KV quantization with selective INT8 protection of attention sinks and recent context. Protecting a bounded number of positions reduces sampler mismatch while retaining most of the speedup from INT4. Across 400 iterations of Qwen3-8B-Base GRPO training, KV4-RL achieves 1.80× cumulative rollout speedup and 1.27× cumulative full-iteration speedup over BF16 rollout, reaching 1.97× rollout speedup over the final 50 iterations. Under a common BF16 evaluator, its final six-benchmark mean accuracy is within 0.5 percentage points of BF16-rollout training for both Qwen3-4B-Base and Qwen3-8B-Base.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in KV cache quantization, one of 12 directions in Efficient LLM inference and serving. Builds on 26 of its 43 references that are in the map. First posted on arXiv 10 Oct 2026.

Builds on

26 of 43 references.

Closest ancestors · 8

Same direction, newest first

  1. 2026SAW-INT4: System-Aware 4-Bit KV-Cache Quantization for Real-World LLM Serving21 Apr
  2. 2025KVSink: Understanding and Enhancing the Preservation of Attention Sinks in KV Cache Quantization for LLMs6 Aug
  3. 2025PM-KVQ: Progressive Mixed-precision KV Cache Quantization for Long-CoT LLMs24 May
  4. 2024SKVQ: Sliding-window Key and Value Cache Quantization for Large Language Models10 May
  5. 2024QServe: W4A8KV4 Quantization and System Co-design for Efficient LLM Serving7 May
  6. 2024QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs30 Mar
  7. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache5 Feb
  8. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization31 Jan

Other references · 18

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  3. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  4. 2023Let's Verify Step by Stepcited by 3,789
  5. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,810
  6. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  7. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
  8. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,392
  9. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,136
  10. 2023PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallelcited by 660
  11. 2025FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Servingcited by 258
  12. 2025AReaL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoningcited by 250
6 more
  1. 2025Insights into DeepSeek-V3: Scaling Challenges and Reflections on Hardware for AI Architecturescited by 39
  2. 2025APRIL: Active Partial Rollouts in Reinforcement Learning to Tame Long-tail Generationcited by 30
  3. 2025QeRL: Beyond Efficiency -- Quantization-enhanced Reinforcement Learning for LLMscited by 16
  4. 2026FP8-RL: A Practical and Stable Low-Precision Stack for LLM Reinforcement Learningcited by 13
  5. 2026StaleFlow: Staleness-Aware Data Management for Mitigating Data Skewness in Fully Disaggregated RL Post-Trainingcited by 11
  6. 2026QUADS: Stabilizing NVFP4 Reinforcement Learning for MoE via QUantization-error Alignment across Dual Sidescited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025BroRL: Scaling Reinforcement Learning via Broadened Exploration
  2. 2026Scheduling Mixed RL Rollouts Beyond Prefix Locality
  3. 2026TRACE: Rollout-Guided Quantization-Aware Training for FP4 Reinforcement Learning of MoE Language Models
  4. 2026S3R: Fast and Faithful RL Rollouts via Self-Sparse Speculative Decoding
  5. 2026QuRL: Efficient Reinforcement Learning with Quantized Rollout
  6. 2026QaRL: Rollout-Aligned Quantization-Aware RL for Fast and Stable Training under Training--Inference Mismatch
  7. 2026Sparse-RL: Breaking the Memory Wall in LLM Reinforcement Learning via Stable Sparse Rollouts
  8. 2026Sparrow: Sparse Rollout for Stable and Efficient Long-context RL of Large Language Models