# Decoupled Networks

arXiv: https://arxiv.org/abs/1804.08071 · HTML: https://mlsanity.com/paper/1804.08071/

## Key facts

- Cited by 45 papers in the map (data through 13 Oct 2026).
- First posted on arXiv 22 Apr 2018.

## Built on by (most cited first, 30 of 45)

- [Controlling Text-to-Image Diffusion by Orthogonal Finetuning](https://mlsanity.com/paper/2306.07280/) (2023)
- [B-cos Networks: Alignment is All We Need for Interpretability](https://mlsanity.com/paper/2205.10268/) (2022)
- [nGPT: Normalized Transformer with Representation Learning on the Hypersphere](https://mlsanity.com/paper/2410.01131/) (2024)
- [Learning with Hyperspherical Uniformity](https://mlsanity.com/paper/2103.01649/) (2021)
- [B-cos Alignment for Inherently Interpretable CNNs and Vision Transformers](https://mlsanity.com/paper/2306.10898/) (2023)
- [Regular Polytope Networks](https://mlsanity.com/paper/2103.15632/) (2021)
- [Learning by Turning: Neural Architecture Aware Optimisation](https://mlsanity.com/paper/2102.07227/) (2021)
- [SphereFace Revived: Unifying Hyperspherical Face Recognition](https://mlsanity.com/paper/2109.05565/) (2021)
- [Maximum Class Separation as Inductive Bias in One Matrix](https://mlsanity.com/paper/2206.08704/) (2022)
- [A Geometric Perspective towards Neural Calibration via Sensitivity Decomposition](https://mlsanity.com/paper/2110.14577/) (2021)
- [Fantastic Pretraining Optimizers and Where to Find Them II: Hyperball Optimization](https://mlsanity.com/paper/2606.16899/) (2026)
- [Generalizing and Decoupling Neural Collapse via Hyperspherical Uniformity Gap](https://mlsanity.com/paper/2303.06484/) (2023)
- [Stronger Normalization-Free Transformers](https://mlsanity.com/paper/2512.10938/) (2025)
- [Continual Learning by Modeling Intra-Class Variation](https://mlsanity.com/paper/2210.05398/) (2022)
- [DeViL: Decoding Vision features into Language](https://mlsanity.com/paper/2309.01617/) (2023)
- [FedUV: Uniformity and Variance for Heterogeneous Federated Learning](https://mlsanity.com/paper/2402.18372/) (2024)
- [Reparameterized LLM Training via Orthogonal Equivalence Transformation](https://mlsanity.com/paper/2506.08001/) (2025)
- [Iterative Teaching by Data Hallucination](https://mlsanity.com/paper/2210.17467/) (2022)
- [Exploring Covariate and Concept Shift for Detection and Calibration of Out-of-Distribution Data](https://mlsanity.com/paper/2110.15231/) (2021)
- [Maximally Compact and Separated Features with Regular Polytope Networks](https://mlsanity.com/paper/2301.06116/) (2023)
- [UER: A Heuristic Bias Addressing Approach for Online Continual Learning](https://mlsanity.com/paper/2309.04081/) (2023)
- [Data-Efficient Learning via Minimizing Hyperspherical Energy](https://mlsanity.com/paper/2206.15204/) (2022)
- [Hyperspherical Quantization: Toward Smaller and More Accurate Models](https://mlsanity.com/paper/2212.12653/) (2022)
- [Model Merging with Functional Dual Anchors](https://mlsanity.com/paper/2510.21223/) (2025)
- [Hyperspherical Classification with Dynamic Label-to-Prototype Assignment](https://mlsanity.com/paper/2403.16937/) (2024)
- [The Compositional Architecture of Regret in Large Language Models](https://mlsanity.com/paper/2506.15617/) (2025)
- [Orthogonal Model Merging](https://mlsanity.com/paper/2602.05943/) (2026)
- [Improving Neural Network Training by Decoupling the Magnitude and Direction of Weight Vectors](https://mlsanity.com/paper/2606.25971/) (2026)
- [Quality-Aware Multimodal Biometric Recognition](https://mlsanity.com/paper/2112.05827/) (2021)
- [Infinite Class Mixup](https://mlsanity.com/paper/2305.10293/) (2023)

## Abstract (arXiv)

Inner product-based convolution has been a central component of convolutional neural networks (CNNs) and the key to learning visual representations. Inspired by the observation that CNN-learned features are naturally decoupled with the norm of features corresponding to the intra-class variation and the angle corresponding to the semantic difference, we propose a generic decoupled learning framework which models the intra-class variation and semantic difference independently. Specifically, we first reparametrize the inner product to a decoupled form and then generalize it to the decoupled convolution operator which serves as the building block of our decoupled networks. We present several effective instances of the decoupled convolution operator. Each decoupled operator is well motivated and has an intuitive geometric interpretation. Based on these decoupled operators, we further propose to directly learn the operator from data. Extensive experiments show that such decoupled reparameterization renders significant performance gain with easier convergence and stronger robustness.