# Regularizing CNNs with Locally Constrained Decorrelations

arXiv: https://arxiv.org/abs/1611.01967 · HTML: https://mlsanity.com/paper/1611.01967/

## Key facts

- Cited by 41 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Nov 2016.

## Built on by (most cited first, 30 of 41)

- [Orthogonalizing Convolutional Layers with the Cayley Transform](https://mlsanity.com/paper/2104.07167/) (2021)
- [OOD-GNN: Out-of-Distribution Generalized Graph Neural Network](https://mlsanity.com/paper/2112.03806/) (2021)
- [Orthogonal Projection Loss](https://mlsanity.com/paper/2103.14021/) (2021)
- [Fast and accurate optimization on the orthogonal manifold without retraction](https://mlsanity.com/paper/2102.07432/) (2021)
- [Learning with Hyperspherical Uniformity](https://mlsanity.com/paper/2103.01649/) (2021)
- [Orthogonal Jacobian Regularization for Unsupervised Disentanglement in Image Generation](https://mlsanity.com/paper/2108.07668/) (2021)
- [The Principle of Diversity: Training Stronger Vision Transformers Calls for Reducing All Levels of Redundancy](https://mlsanity.com/paper/2203.06345/) (2022)
- [Optimization-Induced Graph Implicit Nonlinear Diffusion](https://mlsanity.com/paper/2206.14418/) (2022)
- [Debiased Graph Neural Networks with Agnostic Label Selection Bias](https://mlsanity.com/paper/2201.07708/) (2022)
- [Maximum Class Separation as Inductive Bias in One Matrix](https://mlsanity.com/paper/2206.08704/) (2022)
- [A Survey of Geometric Optimization for Deep Learning: From Euclidean Space to Riemannian Manifold](https://mlsanity.com/paper/2302.08210/) (2023)
- [Feature Overcorrelation in Deep Graph Neural Networks: A New Perspective](https://mlsanity.com/paper/2206.07743/) (2022)
- [Curve Your Enthusiasm: Concurvity Regularization in Differentiable Generalized Additive Models](https://mlsanity.com/paper/2305.11475/) (2023)
- [HOLa: Zero-Shot HOI Detection with Low-Rank Decomposed VLM Feature Adaptation](https://mlsanity.com/paper/2507.15542/) (2025)
- [Momentum Stiefel Optimizer, with Applications to Suitably-Orthogonal Attention, and Optimal Transport](https://mlsanity.com/paper/2205.14173/) (2022)
- [ELRT: Efficient Low-Rank Training for Compact Convolutional Neural Networks](https://mlsanity.com/paper/2401.10341/) (2024)
- [Transfer Learning Gaussian Anomaly Detection by Fine-tuning Representations](https://mlsanity.com/paper/2108.04116/) (2021)
- [Preventing Dimensional Collapse in Self-Supervised Learning via Orthogonality Regularization](https://mlsanity.com/paper/2411.00392/) (2024)
- [OrtSAE: Orthogonal Sparse Autoencoders Uncover Atomic Features](https://mlsanity.com/paper/2509.22033/) (2025)
- [Optimization Induced Equilibrium Networks](https://mlsanity.com/paper/2105.13228/) (2021)
- [Feedback Gradient Descent: Efficient and Stable Optimization with Orthogonality for DNNs](https://mlsanity.com/paper/2205.08385/) (2022)
- [Orthogonal SVD Covariance Conditioning and Latent Disentanglement](https://mlsanity.com/paper/2212.05599/) (2022)
- [Better Training using Weight-Constrained Stochastic Dynamics](https://mlsanity.com/paper/2106.10704/) (2021)
- [Trap of Feature Diversity in the Learning of MLPs](https://mlsanity.com/paper/2112.00980/) (2021)
- [Entangled Residual Mappings](https://mlsanity.com/paper/2206.01261/) (2022)
- [Improving Covariance Conditioning of the SVD Meta-layer by Orthogonality](https://mlsanity.com/paper/2207.02119/) (2022)
- [WLD-Reg: A Data-dependent Within-layer Diversity Regularizer](https://mlsanity.com/paper/2301.01352/) (2023)
- [GSSF: Generalized Structural Sparse Function for Deep Cross-modal Metric Learning](https://mlsanity.com/paper/2410.15266/) (2024)
- [Channel redundancy and overlap in convolutional neural networks with channel-wise NNK graphs](https://mlsanity.com/paper/2110.11400/) (2021)
- [Towards Better Orthogonality Regularization with Disentangled Norm in Training Deep CNNs](https://mlsanity.com/paper/2306.09939/) (2023)

## Abstract (arXiv)

Regularization is key for deep learning since it allows training more complex models while keeping lower levels of overfitting. However, the most prevalent regularizations do not leverage all the capacity of the models since they rely on reducing the effective number of parameters. Feature decorrelation is an alternative for using the full capacity of the models but the overfitting reduction margins are too narrow given the overhead it introduces. In this paper, we show that regularizing negatively correlated features is an obstacle for effective decorrelation and present OrthoReg, a novel regularization technique that locally enforces feature orthogonality. As a result, imposing locality constraints in feature decorrelation removes interferences between negatively correlated feature weights, allowing the regularizer to reach higher decorrelation bounds, and reducing the overfitting more effectively. In particular, we show that the models regularized with OrthoReg have higher accuracy bounds even when batch normalization and dropout are present. Moreover, since our regularization is directly performed on the weights, it is especially suitable for fully convolutional neural networks, where the weight space is constant compared to the feature map space. As a result, we are able to reduce the overfitting of state-of-the-art CNNs on CIFAR-10, CIFAR-100, and SVHN.