mlsanityData to 7 Oct 2026

Hyperspherical Normalization for Scalable Deep Reinforcement Learning

arXiv 2502.15280 ↗First posted 21 Feb 2025cs.LGcited by 35

Not on the map. Why?

Abstract (arXiv)

Scaling up the model size and computation has brought consistent performance improvements in supervised learning. However, this lesson often fails to apply to reinforcement learning (RL) because training the model on non-stationary data easily leads to overfitting and unstable optimization. In response, we introduce SimbaV2, a novel RL architecture designed to stabilize optimization by (i) constraining the growth of weight and feature norm by hyperspherical normalization; and (ii) using a distributional value estimation with reward scaling to maintain stable gradients under varying reward magnitudes. Using the soft actor-critic as a base algorithm, SimbaV2 scales up effectively with larger models and greater compute, achieving state-of-the-art performance on 57 continuous control tasks across 4 domains. The code is available at https://dojeon-ai.github.io/SimbaV2.

Key facts. Cited by 35 papers in the map (data through 7 Oct 2026). Builds on 78 of its 96 references that are in the map. First posted on arXiv 21 Feb 2025.

Builds on

78 of 96 references.

References · 78

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2022Training language models to follow instructions with human feedbackcited by 15,712
  5. 2017Proximal Policy Optimization Algorithmscited by 14,441
  6. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  7. 2020Scaling Laws for Neural Language Modelscited by 6,411
  8. 2016Layer Normalizationcited by 5,579
  9. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  10. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  11. 2015Continuous control with deep reinforcement learningcited by 3,279
  12. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
38 more
  1. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  2. 2016OpenAI Gymcited by 1,684
  3. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  4. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
  5. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,393
  6. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,245
  7. 2023Mastering Diverse Domains through World Modelscited by 1,193
  8. 2018DeepMind Control Suitecited by 935
  9. 2020On Layer Normalization in the Transformer Architecturecited by 906
  10. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 882
  11. 2017A Distributional Perspective on Reinforcement Learningcited by 743
  12. 2017SphereFace: Deep Hypersphere Embedding for Face Recognitioncited by 733
  13. 2017Deep Reinforcement Learning that Matterscited by 727
  14. 2023Scaling Vision Transformers to 22 Billion Parameterscited by 690
  15. 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 573
  16. 2022Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learningcited by 543
  17. 2016Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networkscited by 541
  18. 2023TD-MPC2: Scalable, Robust World Models for Continuous Controlcited by 482
  19. 2019Learning agile and dynamic motor skills for legged robotscited by 402
  20. 2023Analyzing and Improving the Training Dynamics of Diffusion Modelscited by 396
  21. 2023Efficient Online Reinforcement Learning with Offline Datacited by 332
  22. 2023IDQL: Implicit Q-Learning as an Actor-Critic Method with Diffusion Policiescited by 307
  23. 2021Randomized Ensembled Double Q-Learning: Learning Fast Without a Modelcited by 278
  24. 2022The Primacy Bias in Deep Reinforcement Learningcited by 242
  25. 2017NormFace: L2 Hypersphere Embedding for Face Verificationcited by 237
  26. 2023The Dormant Neuron Phenomenon in Deep Reinforcement Learningcited by 171
  27. 2023Understanding plasticity in neural networkscited by 170
  28. 2020Controlling Overestimation Bias with Truncated Mixture of Continuous Distributional Quantile Criticscited by 154
  29. 2023Bigger, Better, Faster: Human-level Atari with human-level efficiencycited by 145
  30. 2022Understanding and Preventing Capacity Loss in Reinforcement Learningcited by 124
  31. 2018Generalization and Regularization in DQNcited by 117
  32. 2023Extreme Q-Learning: MaxEnt RL without Entropycited by 113
  33. 2021Dropout Q-Functions for Doubly Efficient Reinforcement Learningcited by 108
  34. 2024Bigger, Regularized, Optimistic: scaling for compute and sample-efficient continuous controlcited by 104
  35. 2024HumanoidBench: Simulated Humanoid Benchmark for Whole-Body Locomotion and Manipulationcited by 101
  36. 2019Observational Overfitting in Reinforcement Learningcited by 90
  37. 2019CrossQ: Batch Normalization in Deep Reinforcement Learning for Greater Sample Efficiency and Simplicitycited by 89
  38. 2022MyoSuite -- A contact-rich simulation suite for musculoskeletal motor controlcited by 80

50 most cited of 78 All 78 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 10; 2026: 25.

35 papers.

Sort:

By area:

  1. 2025Bigger, Regularized, Categorical: High-Capacity Value Functions are Efficient Multi-Task Learnerscited by 23
  2. 2025floq: Training Critics via Flow-Matching for Scaling Compute in Value-Based RLcited by 20
  3. 2025Measure gradients, not activations! Enhancing neuronal activity in deep reinforcement learningcited by 14
  4. 2025Reparameterized LLM Training via Orthogonal Equivalence Transformationcited by 8
  5. 2026FIRE: Frobenius-Isometry Reinitialization for Balancing the Stability-Plasticity Tradeoffcited by 7
  6. 2025Compute-Optimal Scaling for Value-Based Deep RLcited by 5
  7. 2026The Surprising Difficulty of Search in Model-Based Reinforcement Learningcited by 5
  8. 2026SEAR: Sample Efficient Action Chunking Reinforcement Learningcited by 5
  9. 2026FlashSAC: Fast and Stable Off-Policy Reinforcement Learning for High-Dimensional Robot Controlcited by 5
  10. 2025Bridging the Performance Gap Between Target-Free and Target-Based Reinforcement Learningcited by 4
  11. 2026Scalable Maximum Entropy Reinforcement Learning for Diffusion Policies via Adjoint Matchingcited by 4
  12. 2026What Makes Value Learning Efficient in Residual Reinforcement Learning?cited by 3
23 more
  1. 2026NerVE: Nonlinear Eigenspectrum Dynamics in LLM Feed-Forward Networkscited by 3
  2. 2026Improving Neural Network Training by Decoupling the Magnitude and Direction of Weight Vectorscited by 3
  3. 2025Plasticine: Accelerating Research in Plasticity-Motivated Deep Reinforcement Learningcited by 2
  4. 2025Knowing When to Quit: Probabilistic Early Exits for Speech Separationcited by 2
  5. 2026Squint: Fast Visual Reinforcement Learning for Sim-to-Real Roboticscited by 2
  6. 2026What Does Flow Matching Bring To TD Learning?cited by 2
  7. 2026EfficientTDMPC: Improved MPC Objectives for Sample-Efficient Continuous Controlcited by 2
  8. 2025Rethinking the Role of Dynamic Sparse Training for Scalable Deep Reinforcement Learningcited by 1
  9. 2026RN-D: Discretized Categorical Actors for On-Policy Reinforcement Learningcited by 1
  10. 2026Reward-Conditioned Reinforcement Learningcited by 1
  11. 2026Dream-MPC: Gradient-Based Model Predictive Control with Latent Imaginationcited by 1
  12. 2026Debiased Model-based Representations for Sample-efficient Continuous Controlcited by 1
  13. 2026Behavior-Consistent Deep Reinforcement Learningcited by 1
  14. 2025Scaling DRL for Decision Making: A Survey on Data, Network, and Training Budget Strategiescited by 0
  15. 2026Low-Rank Adaptation for Critic Learning in Off-Policy Reinforcement Learningcited by 0
  16. 2026ANO: Robust Policy Optimization via Bounded, Redescending Gain Fieldscited by 0
  17. 2026Extending Differential Temporal Difference Methods for Episodic Problemscited by 0
  18. 2026When Does Non-Uniform Replay Matter in Reinforcement Learning?cited by 0
  19. 2026R2R2: Robust Representation for Intensive Experience Reuse via Redundancy Reduction in Self-Predictive Learningcited by 0
  20. 2026Beyond Isolation: Unlocking Reinforcement Learning Component Synergy for Sample-Efficient Continuous Controlcited by 0
  21. 2026SingularClip: Preventing Spectral Collapse to Maintain Plasticity in Continual and Reinforcement Learningcited by 0
  22. 2026WarpSAC: Towards the Pinnacle of Scalable Off-policy RL by Rethinking Exploration and Exploitationcited by 0
  23. 2026Mind the Phase: Effective Rank and Representation Health in Legged Locomotioncited by 0