# Decoupled Orthogonal Dynamics: Regularization for Deep Network Optimizers

arXiv: https://arxiv.org/abs/2602.05136 · HTML: https://mlsanity.com/paper/2602.05136/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 5 of its 9 references that are in the map.
- First posted on arXiv 4 Feb 2026.

## Builds on (most cited first, 5 of 5)

- [Visualizing the Loss Landscape of Neural Nets](https://mlsanity.com/paper/1712.09913/) (2017)
- [Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets](https://mlsanity.com/paper/2201.02177/) (2022)
- [A Margin-Maximizing Fine-Grained Ensemble Method](https://mlsanity.com/paper/2409.12849/) (2024)
- [Improving Deep Learning Optimization through Constrained Parameter Regularization](https://mlsanity.com/paper/2311.09058/) (2023)
- [Weight Norm Control](https://mlsanity.com/paper/2311.11446/) (2023)

## Built on by (most cited first, 1 of 1)

- [Nora: Normalized Orthogonal Row Alignment for Scalable Matrix Optimizer](https://mlsanity.com/paper/2605.03769/) (2026)

## Abstract (arXiv)

Is the standard weight decay in AdamW truly optimal? Although AdamW decouples weight decay from adaptive gradient scaling, a fundamental conflict remains: the Radial Tug-of-War. In deep learning, gradients tend to increase parameter norms to expand effective capacity while steering directions to learn features, whereas weight decay indiscriminately suppresses norm growth. This push--pull interaction induces radial oscillations, injecting noise into Adam's second-moment estimates and potentially degrading delicate tangential feature learning. We argue that magnitude and direction play distinct roles and should be decoupled in optimizer dynamics. We propose Orthogonal Dynamics Decoupling and instantiate it as AdamO: an SGD-style update handles the one-dimensional norm control, while Adam's adaptive preconditioning is confined to the tangential subspace. AdamO further incorporates curvature-adaptive radial step sizing and architecture-aware rules and projections for scale-invariant layers and low-dimensional parameters. Experiments on vision and language tasks show that AdamO improves generalization and stability over AdamW without introducing additional complex constraints.