mlsanityData to 14 Oct 2026

WarpSAC: Towards the Pinnacle of Scalable Off-policy RL by Rethinking Exploration and Exploitation

arXiv 2608.24479 ↗First posted 25 Aug 2026cs.LGcited by 0

On the map: Deep reinforcement learning methods → Deep RL algorithms and representations

Abstract (arXiv)

Massively parallel simulation changes the data regime in which off-policy reinforcement learning (RL) is trained, challenging stabilizers designed for data-limited replay. Through controlled experiments across eight benchmark families, we show that these stabilizers are data-regime-dependent: parameter normalization helps with narrow replay coverage but restricts value fitting when data are abundant, while clipped double-Q can be relaxed in high-throughput manipulation. Age-biased replay weighting improves learning efficiency across regimes, especially with limited network capacity. Based on these findings, we propose WarpSAC, a regime-aware family of off-policy RL algorithms. WarpSAC uses Sample Weight Decay for efficient exploitation and provides two variants: WarpSAC-L (Norm ON, clipped double-Q) for data-limited CPU-scale training, and WarpSAC-A (Norm OFF, single-Q) for data-abundant GPU-parallel training. WarpSAC improves normalized score--step AUC over FlashSAC by 4.5% across nine CPU-scale environments and 23.1% across fourteen GPU-parallel environments. It increases UnitreeG1TransportBox-v1 success rate from 19.8% to 96.4%, improves mean normalized wall-time AUC on MuJoCo Playground by 19.1%, and achieves 36.4% faster sim-to-real deployment on Unitree G1 than FlashSAC. These results show that scalable off-policy RL should adapt its stabilizers to the available data regime.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 25 of its 32 references that are in the map. First posted on arXiv 25 Aug 2026.

Builds on

25 of 32 references.

Closest ancestors · 6

Same direction, newest first

  1. 2026FlashSAC: Fast and Stable Off-Policy Reinforcement Learning for High-Dimensional Robot Control6 Apr
  2. 2026The Rank and Gradient Lost in Non-stationarity: Sample Weight Decay for Mitigating Plasticity Loss in Reinforcement Learning2 Apr
  3. 2025XQC: Well-conditioned Optimization Accelerates Deep Reinforcement Learning29 Sep
  4. 2025Mitigating Plasticity Loss in Continual Reinforcement Learning by Reducing Churn31 May
  5. 2025Hyperspherical Normalization for Scalable Deep Reinforcement Learning21 Feb
  6. 2024Improving Deep Reinforcement Learning by Reducing the Chain Effect of Value and Policy Churn7 Sep

Other references · 19

  1. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  2. 2016Layer Normalizationcited by 5,669
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  4. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  5. 2018Spectral Normalization for Generative Adversarial Networkscited by 1,636
  6. 2019Root Mean Square Layer Normalizationcited by 1,480
  7. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,240
  8. 2015Prioritized Experience Replaycited by 1,099
  9. 2018DeepMind Control Suitecited by 956
  10. 2017Spectrally-normalized margin bounds for neural networkscited by 686
  11. 2016Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networkscited by 543
  12. 2023TD-MPC2: Scalable, Robust World Models for Continuous Controlcited by 512
7 more
  1. 2021Randomized Ensembled Double Q-Learning: Learning Fast Without a Modelcited by 288
  2. 2022The Primacy Bias in Deep Reinforcement Learningcited by 247
  3. 2024ManiSkill3: GPU Parallelized Robotics Simulation and Rendering for Generalizable Embodied AIcited by 229
  4. 2023The Dormant Neuron Phenomenon in Deep Reinforcement Learningcited by 174
  5. 2014Model-based Reinforcement Learning and the Eluder Dimensioncited by 116
  6. 2024HumanoidBench: Simulated Humanoid Benchmark for Whole-Body Locomotion and Manipulationcited by 112
  7. 2022MyoSuite -- A contact-rich simulation suite for musculoskeletal motor controlcited by 80

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026CODS: Iterative Bellman-Residual Data Selection for Reusable Offline Reinforcement Learning
  2. 2024Offline-Boosted Actor-Critic: Adaptively Blending Optimal Historical Behaviors in Deep Off-Policy RL
  3. 2026Enabling Token-wise Importance Sampling in Off-Policy RLVR with Proximal KL Regularization
  4. 2026SOPE: Stabilizing Off-Policy Evaluation for Online RL with Prior Data
  5. 2026Reuse your FLOPs: Scaling RL on Hard Problems by Conditioning on Very Off-Policy Prefixes
  6. 2023Finetuning from Offline Reinforcement Learning: Challenges, Trade-offs and Practical Solutions
  7. 2025Staggered Environment Resets Improve Massively Parallel On-Policy Reinforcement Learning
  8. 2025A Clean Slate for Offline Reinforcement Learning