mlsanityData to 14 Oct 2026

XQC: Well-conditioned Optimization Accelerates Deep Reinforcement Learning

arXiv 2509.25174 ↗First posted 29 Sep 2025cs.LG cs.AIcited by 17

On the map: Deep reinforcement learning methods → Deep RL algorithms and representations

Abstract (arXiv)

Sample efficiency is a central property of effective deep reinforcement learning algorithms. Recent work has improved this through added complexity, such as larger models, exotic network architectures, and more complex algorithms, which are typically motivated purely by empirical performance. We take a more principled approach by focusing on the optimization landscape of the critic network. Using the eigenspectrum and condition number of the critic's Hessian, we systematically investigate the impact of common architectural design decisions on training dynamics. Our analysis reveals that a novel combination of batch normalization (BN), weight normalization (WN), and a distributional cross-entropy (CE) loss produces condition numbers orders of magnitude smaller than baselines. This combination also naturally bounds gradient norms, a property critical for maintaining a stable effective learning rate under non-stationary targets and bootstrapping. Based on these insights, we introduce XQC: a well-motivated, sample-efficient deep actor-critic algorithm built upon soft actor-critic that embodies these optimization-aware principles. We achieve state-of-the art sample efficiency across 55 proprioception and 15 vision-based continuous control tasks, all while using significantly fewer parameters than competing methods. Our code is available at danielpalenicek.github.io/projects/xqc.

Key facts. Cited by 17 papers in the map (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 39 of its 56 references that are in the map. First posted on arXiv 29 Sep 2025.

Builds on

39 of 56 references.

Closest ancestors · 13

Same direction, newest first

  1. 2025Compute-Optimal Scaling for Value-Based Deep RL20 Aug
  2. 2025A Forget-and-Grow Strategy for Deep Reinforcement Learning Scaling in Continuous Control3 Jul
  3. 2025Network Sparsity Unlocks the Scaling Potential of Deep Reinforcement Learning20 Jun
  4. 2025Stable Gradients for Stable Learning at Scale in Deep Reinforcement Learning18 Jun
  5. 2025Scaling Off-Policy Reinforcement Learning with Batch and Weight Normalization11 Feb
  6. 2025Value-Based Deep RL Scales Predictably6 Feb
  7. 2025Towards General-Purpose Model-Free Reinforcement Learning27 Jan
  8. 2024SimBa: Simplicity Bias for Scaling Up Parameters in Deep Reinforcement Learning13 Oct
  9. 2024Neuroplastic Expansion in Deep Reinforcement Learning10 Oct
  10. 2024Normalization and effective learning rates in reinforcement learning1 Jul
  11. 2024Bigger, Regularized, Optimistic: scaling for compute and sample-efficient continuous control25 May
  12. 2024Dissecting Deep RL with High Update Ratios: Combatting Value Divergence9 Mar
1 more
  1. 2024Stop Regressing: Training Value Functions via Classification for Scalable Deep RL6 Mar

Other references · 26

  1. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  2. 2016Layer Normalizationcited by 5,669
  3. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,653
  4. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,427
  5. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,240
  6. 2018DeepMind Control Suitecited by 956
  7. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 807
  8. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 773
  9. 2017A Distributional Perspective on Reinforcement Learningcited by 765
  10. 2023TD-MPC2: Scalable, Robust World Models for Continuous Controlcited by 512
  11. 2023Efficient Online Reinforcement Learning with Offline Datacited by 363
  12. 2019An Investigation into Neural Net Optimization via Hessian Eigenvalue Densitycited by 292
14 more
  1. 2021Randomized Ensembled Double Q-Learning: Learning Fast Without a Modelcited by 288
  2. 2022The Primacy Bias in Deep Reinforcement Learningcited by 247
  3. 2020Implementation Matters in Deep Policy Gradients: A Case Study on PPO and TRPOcited by 186
  4. 2017L2 Regularization versus Batch and Weight Normalizationcited by 124
  5. 2021Dropout Q-Functions for Doubly Efficient Reinforcement Learningcited by 114
  6. 2024HumanoidBench: Simulated Humanoid Benchmark for Whole-Body Locomotion and Manipulationcited by 112
  7. 2019CrossQ: Batch Normalization in Deep Reinforcement Learning for Greater Sample Efficiency and Simplicitycited by 95
  8. 2022MyoSuite -- A contact-rich simulation suite for musculoskeletal motor controlcited by 80
  9. 2024nGPT: Normalized Transformer with Representation Learning on the Hyperspherecited by 70
  10. 2018Improving Regression Performance with Distributional Lossescited by 67
  11. 2025Bigger, Regularized, Categorical: High-Capacity Value Functions are Efficient Multi-Task Learnerscited by 23
  12. 2023Coherent Soft Imitation Learningcited by 22
  13. 2025Gait in Eight: Efficient On-Robot Learning for Omnidirectional Quadruped Locomotioncited by 8
  14. 2023Diminishing Return of Value Expansion Methods in Model-Based Reinforcement Learningcited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 16.

17 papers.

Sort:

By area:

  1. 2026FlashSAC: Fast and Stable Off-Policy Reinforcement Learning for High-Dimensional Robot Controlcited by 5
  2. 2026Scalable Maximum Entropy Reinforcement Learning for Diffusion Policies via Adjoint Matchingcited by 5
  3. 2025Use the Online Network If You Can: Towards Fast and Stable Reinforcement Learningcited by 2
  4. 2026What Does Flow Matching Bring To TD Learning?cited by 2
  5. 2026Reward-Conditioned Reinforcement Learningcited by 1
  6. 2026Revisiting Adam for Streaming Reinforcement Learningcited by 1
  7. 2026XQCfD: Accelerating Fast Actor-Critic Algorithms with Prior Data and Prior Policiescited by 1
  8. 2026Behavior-Consistent Deep Reinforcement Learningcited by 1
  9. 2026Gradient Iterated Temporal-Difference Learningcited by 0
  10. 2026When Does Non-Uniform Replay Matter in Reinforcement Learning?cited by 0
  11. 2026Trust Region Inverse Reinforcement Learning: Explicit Dual Ascent using Local Policy Updatescited by 0
  12. 2026Coherent Off-Policy Improvement of Large Behavior Models with Learned Rewardscited by 0
5 more
  1. 2026Momba: Network Modernization Improves Multi-Objective Reinforcement Learningcited by 0
  2. 2026SingularClip: Preventing Spectral Collapse to Maintain Plasticity in Continual and Reinforcement Learningcited by 0
  3. 2026WarpSAC: Towards the Pinnacle of Scalable Off-policy RL by Rethinking Exploration and Exploitationcited by 0
  4. 2026On BatchNorm Forward Modes in Value-Based Reinforcement Learningcited by 0
  5. 2026ChronoSRL: Temporal Geometry for Self-Supervised Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025An Investigation of Batch Normalization in Off-Policy Actor-Critic Algorithms
  2. 2024Quality-Diversity Actor-Critic: Learning High-Performing and Diverse Behaviors via Value and Successor Features Critics
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
  4. 2026ReBRAC-v2: The Return of the King
  5. 2026Actor-Critic with Active Importance Sampling
  6. 2024Risk-Sensitive Soft Actor-Critic for Robust Deep Reinforcement Learning under Distribution Shifts
  7. 2025Enhancing Deep Deterministic Policy Gradients on Continuous Control Tasks with Decoupled Prioritized Experience Replay
  8. 2025Simplicial Embeddings Improve Sample Efficiency in Actor-Critic Agents