mlsanityData to 14 Oct 2026

Clustering-Based Weight Orthogonalization for Stabilizing Deep Reinforcement Learning

arXiv 2511.11607 ↗First posted 2 Nov 2025cs.LG cs.AIcited by 2

On the map: Deep reinforcement learning methods → Deep RL algorithms and representations

Abstract (arXiv)

Reinforcement learning (RL) has made significant advancements, achieving superhuman performance in various tasks. However, RL agents often operate under the assumption of environmental stationarity, which poses a great challenge to learning efficiency since many environments are inherently non-stationary. This non-stationarity results in the requirement of millions of iterations, leading to low sample efficiency. To address this issue, we introduce the Clustering Orthogonal Weight Modified (COWM) layer, which can be integrated into the policy network of any RL algorithm and mitigate non-stationarity effectively. The COWM layer stabilizes the learning process by employing clustering techniques and a projection matrix. Our approach not only improves learning speed but also reduces gradient interference, thereby enhancing the overall learning efficiency. Empirically, the COWM outperforms state-of-the-art methods and achieves improvements of 9% and 12.6% in vision based and state-based DMControl benchmark. It also shows robustness and generality across various algorithms and tasks.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 40 of its 63 references that are in the map. First posted on arXiv 2 Nov 2025.

Builds on

40 of 63 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Sharing Knowledge in Multi-Task Deep Reinforcement Learning17 Jan

Other references · 39

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  3. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,897
  4. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  5. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  6. 2015Continuous control with deep reinforcement learningcited by 3,407
  7. 2015Trust Region Policy Optimizationcited by 2,580
  8. 2016Learning without Forgettingcited by 2,496
  9. 2017Gradient Episodic Memory for Continual Learningcited by 1,898
  10. 2020Gradient Surgery for Multi-Task Learningcited by 1,256
  11. 2018DeepMind Control Suitecited by 956
  12. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 760
27 more
  1. 2017Lifelong Learning with Dynamically Expandable Networkscited by 694
  2. 2015Policy Distillationcited by 440
  3. 2019Learn to Grow: A Continual Structure Learning Framework for Overcoming Catastrophic Forgettingcited by 302
  4. 2020Towards Continual Reinforcement Learning: A Review and Perspectivescited by 274
  5. 2015The Benefit of Multitask Representation Learningcited by 195
  6. 2018Continual Learning of Context-dependent Processing in Neural Networkscited by 179
  7. 2020Conditional Channel Gated Networks for Task-Aware Continual Learningcited by 115
  8. 2017Incremental Learning Through Deep Adaptationcited by 108
  9. 2020Context-aware Dynamics Model for Generalization in Model-Based Reinforcement Learningcited by 98
  10. 2020Off-Dynamics Reinforcement Learning: Training for Transfer with Domain Classifierscited by 97
  11. 2023Video Prediction Models as Rewards for Reinforcement Learningcited by 78
  12. 2013Hidden Parameter Markov Decision Processes: A Semiparametric Regression Approach for Discovering Latent Task Parametrizationscited by 70
  13. 2022When to Trust Your Simulator: Dynamics-Aware Hybrid Offline-and-Online Reinforcement Learningcited by 60
  14. 2019Learning Fast Adaptation with Meta Strategy Optimizationcited by 48
  15. 2020Reparameterizing Convolutions for Incremental Multi-Task Learning without Task Interferencecited by 47
  16. 2018Pseudo-Rehearsal: Achieving Deep Reinforcement Learning without Catastrophic Forgettingcited by 46
  17. 2018Policy Transfer with Strategy Optimizationcited by 43
  18. 2022DARA: Dynamics-Aware Reward Augmentation in Offline Reinforcement Learningcited by 41
  19. 2019Environment Probing Interaction Policiescited by 39
  20. 2023Efficient Deep Reinforcement Learning Requires Regulating Overfittingcited by 36
  21. 2021Meta-Adaptive Nonlinear Control: Theory and Algorithmscited by 23
  22. 2018Organizing Experience: A Deeper Look at Replay Mechanisms for Sample-based Planning in Continuous State Domainscited by 19
  23. 2023Image Augmentation with Controlled Diffusion for Weakly-Supervised Semantic Segmentationcited by 19
  24. 2024Image Fusion for Cross-Domain Sequential Recommendationcited by 17
  25. 2024Image Augmentation Agent for Weakly Supervised Semantic Segmentationcited by 15
  26. 2022Meta-Gradients in Non-Stationary Environmentscited by 11
  27. 2023Differentially Encoded Observation Spaces for Perceptive Reinforcement Learningcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Orthogonal Weight Modification Enhances Learning Scalability and Convergence Efficiency without Gradient Backpropagationcited by 1
  2. 2026StegaFFD: Privacy-Preserving Face Forgery Detection via Fine-Grained Steganographic Domain Liftingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Learning in Low-Dimensional Subspaces: Orthogonal Bottlenecks for Reinforcement Learning
  2. 2025Stabilizing Reinforcement Learning with LLMs: Formulation and Practices
  3. 2025Stable Gradients for Stable Learning at Scale in Deep Reinforcement Learning
  4. 2024Discovering Behavioral Modes in Deep Reinforcement Learning Policies Using Trajectory Clustering in Latent Space
  5. 2022Deep Reinforcement Learning with Vector Quantized Encoding
  6. 2024Implementing Online Reinforcement Learning with Clustering Neural Networks
  7. 2024Enhancing Interpretability in Deep Reinforcement Learning through Semantic Clustering
  8. 2026The Rank and Gradient Lost in Non-stationarity: Sample Weight Decay for Mitigating Plasticity Loss in Reinforcement Learning