Rank Collapse Is Recoverable, Growing |Q| Is Not: Out-of-Sample Early Warning for Value Divergence in High-UTD Soft Actor-Critic
On the map: Deep reinforcement learning methods → RL theory and sample complexity
Abstract (arXiv)
Raising the update-to-data (UTD) ratio breaks off-policy critics in two ways grouped as "plasticity loss": collapsing representations and growing value magnitude |Q|. We separate them in Soft Actor-Critic (SAC) with scaled critics (width 2048, no normalization). Collapse is survivable: at UTD ratio 16, HalfCheetah critics with most units dormant keep learning, and the training guard, which stops runs whose loss or |Q| explodes, never flags them. Within one high-UTD SAC configuration, runs start close together, and how far a critic's log10|Q| has climbed by step 15k, its early growth, ranks the runs by how soon the guard flags them. At 15k, a flagged run's |Q| sits a median of over a hundredfold below its flag level, yet the climb's rate already orders the flags (Harrell's C and out-of-sample AUC 0.78 on Walker2d, 0.98 on Ant, at UTD ratio 4). Dormancy does not. Aborting on this rate saves about a tenth of held-out Walker2d compute and stays net-positive live. A LayerNorm critic lowers the rate, removes the flag on Walker2d at UTD ratio 4 and lowers the return.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 41 of its 60 references that are in the map. First posted on arXiv 26 Sep 2026.
Builds on
41 of 60 references.
Closest ancestors · 1
Same direction, newest first
Other references · 40
- 2016Layer Normalizationcited by 5,669
- 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
- 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
- 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 807
- 2016Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimizationcited by 699
- 2023Efficient Online Reinforcement Learning with Offline Datacited by 363
- 2021Randomized Ensembled Double Q-Learning: Learning Fast Without a Modelcited by 288
- 2022The Primacy Bias in Deep Reinforcement Learningcited by 247
- 2018A System for Massively Parallel Hyperparameter Tuningcited by 202
- 2023Small-scale proxies for large-scale Transformer training instabilitiescited by 183
- 2018Deep Reinforcement Learning and the Deadly Triadcited by 174
- 2023The Dormant Neuron Phenomenon in Deep Reinforcement Learningcited by 174
28 more
- 2023Understanding plasticity in neural networkscited by 174
- 2020Implicit Under-Parameterization Inhibits Data-Efficient Deep Reinforcement Learningcited by 134
- 2022Understanding and Preventing Capacity Loss in Reinforcement Learningcited by 125
- 2023Loss of Plasticity in Continual Deep Reinforcement Learningcited by 121
- 2021Dropout Q-Functions for Doubly Efficient Reinforcement Learningcited by 114
- 2024Bigger, Regularized, Optimistic: scaling for compute and sample-efficient continuous controlcited by 108
- 2019CrossQ: Batch Normalization in Deep Reinforcement Learning for Greater Sample Efficiency and Simplicitycited by 95
- 2024Disentangling the Causes of Plasticity Loss in Neural Networkscited by 86
- 2024SimBa: Simplicity Bias for Scaling Up Parameters in Deep Reinforcement Learningcited by 76
- 2023Deep Reinforcement Learning with Plasticity Injectioncited by 74
- 2023Maintaining Plasticity in Continual Learning via Regenerative Regularizationcited by 65
- 2019Towards Characterizing Divergence in Deep Q-Learningcited by 43
- 2021Is High Variance Unavoidable in RL? A Case Study in Continuous Controlcited by 28
- 2024A Study of Plasticity Loss in On-Policy Deep Reinforcement Learningcited by 26
- 2022An Empirical Study of Implicit Regularization in Deep Offline RLcited by 24
- 2024Plasticity Loss in Deep Reinforcement Learning: A Surveycited by 24
- 2025Stable Gradients for Stable Learning at Scale in Deep Reinforcement Learningcited by 23
- 2025Mitigating Plasticity Loss in Continual Reinforcement Learning by Reducing Churncited by 22
- 2023Understanding, Predicting and Better Resolving Q-Value Divergence in Offline-RLcited by 20
- 2025Scaling Off-Policy Reinforcement Learning with Batch and Weight Normalizationcited by 19
- 2024Dissecting Deep RL with High Update Ratios: Combatting Value Divergencecited by 17
- 2024An Introduction to Deep Survival Analysis Models for Predicting Time-to-Event Outcomescited by 16
- 2024MAD-TD: Model-Augmented Data stabilizes High Update Ratio RLcited by 16
- 2025Value-Based Deep RL Scales Predictablycited by 15
- 2025Network Sparsity Unlocks the Scaling Potential of Deep Reinforcement Learningcited by 11
- 2022Universal Early Warning Signals of Phase Transitions in Climate Systemscited by 10
- 2025Spectral Collapse Drives Loss of Plasticity in Deep Continual Learningcited by 6
- 2026Ultra-Early Prediction of Tipping Points: Integrating Dynamical Measures with Reservoir Computingcited by 2
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025Optimistic critics can empower small actors
- 2026Simple Actors and Deep Critics for Scalable Reinforcement Learning
- 2026Refined Analysis of Entropy-Regularized Actor-Critic
- 2026CARE-VI: Conservative Adaptive Reliability Estimation for Value Improvement in Off-Policy Actor-Critic Learning
- 2025Functional Critics Are Essential for Actor-Critic: From Off-Policy Stability to Efficient Exploration
- 2024No Representation, No Trust: Connecting Representation, Collapse, and Trust Issues in PPO
- 2026When Learning Rates Go Wrong: Early Structural Signals in PPO Actor-Critic
- 2026Unthrottling the Tanh Jacobian in SAC: A Negative Result on Bang-Bang Control and MetaDrive