mlsanityData to 14 Oct 2026

Spectral Normalisation for Deep Reinforcement Learning: an Optimisation Perspective

arXiv 2105.05246 ↗First posted 11 May 2021cs.LG cs.AIcited by 65

Not on the map. Why?

Cited by 24 in Deep reinforcement learning methods, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Most of the recent deep reinforcement learning advances take an RL-centric perspective and focus on refinements of the training objective. We diverge from this view and show we can recover the performance of these developments not by changing the objective, but by regularising the value-function estimator. Constraining the Lipschitz constant of a single layer using spectral normalisation is sufficient to elevate the performance of a Categorical-DQN agent to that of a more elaborated rainbow agent on the challenging Atari domain. We conduct ablation studies to disentangle the various effects normalisation has on the learning dynamics and show that is sufficient to modulate the parameter updates to recover most of the performance of spectral normalisation. These findings hint towards the need to also focus on the neural component and its learning dynamics to tackle the peculiarities of Deep Reinforcement Learning.

Key facts. Cited by 65 papers in the map (data through 14 Oct 2026). Builds on 41 of its 52 references that are in the map. First posted on arXiv 11 May 2021.

Builds on

41 of 52 references.

References · 41

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  3. 2017Improved Training of Wasserstein GANscited by 2,235
  4. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,745
  5. 2018Spectral Normalization for Generative Adversarial Networkscited by 1,636
  6. 2015Prioritized Experience Replaycited by 1,099
  7. 2015Optimizing Neural Networks with Kronecker-factored Approximate Curvaturecited by 796
  8. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 786
  9. 2015Dueling Network Architectures for Deep Reinforcement Learningcited by 785
  10. 2017A Distributional Perspective on Reinforcement Learningcited by 765
  11. 2017Deep Reinforcement Learning that Matterscited by 754
  12. 2020MOPO: Model-based Offline Policy Optimizationcited by 636
29 more
  1. 2016Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networkscited by 543
  2. 2016Deep Exploration via Bootstrapped DQNcited by 527
  3. 2018Lipschitz regularity of deep neural networks: analysis and efficient estimationcited by 415
  4. 2020Simple and Principled Uncertainty Estimation with Deterministic Deep Learning via Distance Awarenesscited by 389
  5. 2018How Does Batch Normalization Help Optimization?cited by 357
  6. 2018Quantifying Generalization in Reinforcement Learningcited by 352
  7. 2018Regularisation of Neural Networks by Enforcing Lipschitz Continuitycited by 314
  8. 2017Parseval Networks: Improving Robustness to Adversarial Examplescited by 293
  9. 2017Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agentscited by 269
  10. 2017Noisy Networks for Explorationcited by 266
  11. 2019Limitations of the Empirical Fisher Approximation for Natural Gradient Descentcited by 217
  12. 2018Sorting out Lipschitz function approximationcited by 213
  13. 2018Lipschitz-Margin Training: Scalable Certification of Perturbation Invariance for Deep Neural Networkscited by 195
  14. 2017Spectral Norm Regularization for Improving the Generalizability of Deep Learningcited by 182
  15. 2013Revisiting Natural Gradient for Deep Networkscited by 159
  16. 2020Implicit Under-Parameterization Inhibits Data-Efficient Deep Reinforcement Learningcited by 134
  17. 2017L2 Regularization versus Batch and Weight Normalizationcited by 124
  18. 2018Dopamine: A Research Framework for Deep Reinforcement Learningcited by 122
  19. 2019Distilling Policy Distillationcited by 84
  20. 2020Revisiting Rainbow: Promoting more Insightful and Inclusive Deep Reinforcement Learning Researchcited by 78
  21. 2018Generalizable Adversarial Training via Spectral Normalizationcited by 68
  22. 2018Theoretical Analysis of Auto Rate-Tuning by Batch Normalizationcited by 65
  23. 2018Norm matters: efficient and accurate normalization schemes in deep networkscited by 62
  24. 2018Limitations of the Lipschitz constant as a defense against adversarial examplescited by 49
  25. 2020A case for new neural network smoothness constraintscited by 43
  26. 2020Interference and Generalization in Temporal Difference Learningcited by 42
  27. 2018Where Did My Optimum Go?: An Empirical Analysis of Gradient Descent Optimization in Policy Gradient Methodscited by 17
  28. 2020Adaptive Temporal Difference Learning with Linear Function Approximationcited by 9
  29. 2020TDprop: Does Jacobi Preconditioning Help Temporal Difference Learning?cited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 5; 2022: 11; 2023: 14; 2024: 9; 2025: 12; 2026: 14.

65 papers.

Sort:

By area:

  1. 2023Understanding plasticity in neural networkscited by 174
  2. 2022Pessimistic Bootstrapping for Uncertainty-Driven Offline Reinforcement Learningcited by 125
  3. 2022Understanding and Preventing Capacity Loss in Reinforcement Learningcited by 125
  4. 2022RORL: Robust Offline Reinforcement Learning via Conservative Smoothingcited by 84
  5. 2024SimBa: Simplicity Bias for Scaling Up Parameters in Deep Reinforcement Learningcited by 76
  6. 2023Deep Reinforcement Learning with Plasticity Injectioncited by 74
  7. 2023Pgx: Hardware-Accelerated Parallel Game Simulators for Reinforcement Learningcited by 51
  8. 2024Overestimation, Overfitting, and Plasticity in Actor-Critic: the Bitter Lesson of Reinforcement Learningcited by 50
  9. 2023PLASTIC: Improving Input and Label Plasticity for Sample Efficient Reinforcement Learningcited by 42
  10. 2022Why neural networks find simple solutions: the many regularizers of geometric complexitycited by 39
  11. 2025Hyperspherical Normalization for Scalable Deep Reinforcement Learningcited by 38
  12. 2023Efficient Deep Reinforcement Learning Requires Regulating Overfittingcited by 36
38 more
  1. 2023Small batch deep reinforcement learningcited by 36
  2. 2022Stabilizing Off-Policy Deep Reinforcement Learning from Pixelscited by 32
  3. 2021Is High Variance Unavoidable in RL? A Case Study in Continuous Controlcited by 28
  4. 2022Hyperbolic Deep Reinforcement Learningcited by 25
  5. 2024Plasticity Loss in Deep Reinforcement Learning: A Surveycited by 24
  6. 2025Stable Gradients for Stable Learning at Scale in Deep Reinforcement Learningcited by 23
  7. 2025Mitigating Plasticity Loss in Continual Reinforcement Learning by Reducing Churncited by 22
  8. 2024On the consistency of hyper-parameter selection in value-based deep reinforcement learningcited by 21
  9. 2025Training Transformers with Enforced Lipschitz Constantscited by 20
  10. 2021Towards Deeper Deep Reinforcement Learning with Spectral Normalizationcited by 19
  11. 2023Is Model Ensemble Necessary? Model-based RL via a Single Model with Lipschitz Regularized Value Functioncited by 19
  12. 2021Functional Regularization for Reinforcement Learning via Learned Fourier Featurescited by 17
  13. 2023Normalization Enhances Generalization in Visual Reinforcement Learningcited by 13
  14. 2025Do We Always Need the Simplicity Bias? Looking for Optimal Inductive Biases in the Wildcited by 13
  15. 2025Simplicial Embeddings Improve Sample Efficiency in Actor-Critic Agentscited by 12
  16. 2021Fast and Data-Efficient Training of Rainbow: an Experimental Study on Ataricited by 10
  17. 2023ReLU to the Rescue: Improve Your On-Policy Actor-Critic with Positive Advantagescited by 10
  18. 2023Learning Multi-Agent Communication with Contrastive Learningcited by 9
  19. 2023Mind the Model, Not the Agent: The Primacy Bias in Model-based RLcited by 9
  20. 2025Advancing Intelligent Sequence Modeling: Evolution, Trade-offs, and Applications of State-Space Architectures from S4 to Mambacited by 9
  21. 2024Beyond The Rainbow: High Performance Deep Reinforcement Learning on a Desktop PCcited by 7
  22. 2022Effects of Spectral Normalization in Multi-agent Reinforcement Learningcited by 6
  23. 2023On the Theory of Risk-Aware Agents: Bridging Actor-Critic and Economicscited by 6
  24. 2025Activation by Interval-wise Dropout: A Simple Way to Prevent Neural Networks from Plasticity Losscited by 6
  25. 2021Maximum Entropy Model-based Reinforcement Learningcited by 5
  26. 2022Exact Spectral Norm Regularization for Neural Networkscited by 5
  27. 2022Learning to Ground Decentralized Multi-Agent Communication with Contrastive Learningcited by 4
  28. 2022How Does Return Distribution in Distributional Reinforcement Learning Help Optimization?cited by 4
  29. 2024Robustness and Generalization in Quantum Reinforcement Learning via Lipschitz Regularizationcited by 4
  30. 2025Evidence on the Regularisation Properties of Maximum-Entropy Reinforcement Learningcited by 3
  31. 2025Contextual Similarity Distillation: Ensemble Uncertainties with a Single Modelcited by 3
  32. 2026Enhancing Control Policy Smoothness by Aligning Actions with Predictions from Preceding Statescited by 3
  33. 2022Robust Offline Reinforcement Learning with Gradient Penalty and Constraint Relaxationcited by 2
  34. 2024The Role of Deep Learning Regularizations on Actors in Offline RLcited by 2
  35. 2026Stabilizing the Q-Gradient Field for Policy Smoothness in Actor-Critic Methodscited by 2
  36. 2026Temporal Difference Learning with Constrained Initial Representationscited by 2
  37. 2023The Ladder in Chaos: A Simple and Effective Improvement to General DRL Algorithms by Policy Path Trimming and Boostingcited by 1
  38. 2025Recovering Plasticity of Neural Networks via Soft Weight Rescalingcited by 1

50 most cited of 65 All 65 (JSON)