Data to 14 Oct 2026

Reducing Overfitting in Deep Networks by Decorrelating Representations

arXiv 1511.06068 ↗First posted 19 Nov 2015cs.LG stat.MLcited by 94

Not on the map. Why?

Abstract (arXiv)

One major challenge in training Deep Neural Networks is preventing overfitting. Many techniques such as data augmentation and novel regularizers such as Dropout have been proposed to prevent overfitting without requiring a massive amount of training data. In this work, we propose a new regularizer called DeCov which leads to significantly reduced overfitting (as indicated by the difference between train and val performance), and better generalization. Our regularizer encourages diverse or non-redundant representations in Deep Neural Networks by minimizing the cross-covariance of hidden activations. This simple intuition has been explored in a number of past works but surprisingly has never been applied as a regularizer in supervised learning. Experiments across a range of datasets and network architectures show that this loss always reduces overfitting while almost always maintaining or increasing generalization performance and often improving performance over Dropout.

Key facts. Cited by 94 papers in the map (data through 14 Oct 2026). First posted on arXiv 19 Nov 2015.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 18; 2022: 19; 2023: 24; 2024: 11; 2025: 9; 2026: 13.

94 papers.

Sort:

By area:

2021On Feature Decorrelation in Self-Supervised Learningcited by 160
2021OOD-GNN: Out-of-Distribution Generalized Graph Neural Networkcited by 78
2021Mimicking the Oracle: An Initial Phase Decorrelation Approach for Class Incremental Learningcited by 49
2022Towards Understanding and Mitigating Dimensional Collapse in Heterogeneous Federated Learningcited by 46
2023In Search of Lost Online Test-time Adaptation: A Surveycited by 44
2022Almost-Orthogonal Layers for Efficient General-Purpose Lipschitz Networkscited by 39
2021Sparse-Interest Network for Sequential Recommendationcited by 38
2021Learning with Hyperspherical Uniformitycited by 35
2024RealViformer: Investigating Attention for Real-World Video Super-Resolutioncited by 33
2022Improving Multi-Task Generalization via Regularizing Spurious Correlationcited by 28
2024AZ-NAS: Assembling Zero-Cost Proxies for Network Architecture Searchcited by 28
2021A Cluster-based Approach for Improving Isotropy in Contextual Embedding Spacecited by 27

38 more

2022Debiased Graph Neural Networks with Agnostic Label Selection Biascited by 25
2024Towards Large-Scale Training of Pathology Foundation Modelscited by 23
2022Weakly-Supervised Semantic Segmentation with Visual Words Learning and Hybrid Poolingcited by 19
2022Feature Overcorrelation in Deep Graph Neural Networks: A New Perspectivecited by 19
2021Knowledge Evolution in Neural Networkscited by 18
2023Randomized Sparse Neural Galerkin Schemes for Solving Evolution Equations with Deep Networkscited by 18
2022Learning Cross-view Geo-localization Embeddings via Dynamic Weighted Decorrelation Regularizationcited by 17
2023VNE: An Effective Method for Improving Deep Representation by Manipulating Eigenvalue Distributioncited by 17
2023Generalizing and Decoupling Neural Collapse via Hyperspherical Uniformity Gapcited by 14
2023Variance-Covariance Regularization Improves Representation Learningcited by 14
2023Regularization of polynomial networks for image recognitioncited by 13
2022Riemannian Hamiltonian methods for min-max optimization on manifoldscited by 12
2023Curve Your Enthusiasm: Concurvity Regularization in Differentiable Generalized Additive Modelscited by 12
2023Interpretable and intervenable ultrasonography-based machine learning models for pediatric appendicitiscited by 11
2023Rec4Ad: A Free Lunch to Mitigate Sample Selection Bias for Ads CTR Prediction in Taobaocited by 11
2022Variance Covariance Regularization Enforces Pairwise Independence in Self-Supervised Representationscited by 10
2021Stochastic Whitening Batch Normalizationcited by 9
2022Constrained Parameter Inference as a Principle for Learningcited by 9
2023On the Importance of Feature Decorrelation for Unsupervised Representation Learning in Reinforcement Learningcited by 8
2025OrtSAE: Orthogonal Sparse Autoencoders Uncover Atomic Featurescited by 8
2022A Detailed Study of Interpretability of Deep Neural Network based Top Taggerscited by 7
2021Unsupervised Information Obfuscation for Split Inference of Neural Networkscited by 5
2022Reducing Redundancy in the Bottleneck Representation of the Autoencoderscited by 5
2022FeaRLESS: Feature Refinement Loss for Ensembling Self-Supervised Learning Features in Robust End-to-end Speech Recognitioncited by 5
2023Diffused Redundancy in Pre-trained Representationscited by 5
2023Hierarchical Semantic Tree Concept Whitening for Interpretable Image Classificationcited by 5
2025Rethinking Layer-wise Model Merging through Chain of Mergescited by 5
2021Learning distinct features helps, provablycited by 4
2021Encouraging Intra-Class Diversity Through a Reverse Contrastive Loss for Better Single-Source Domain Generalizationcited by 4
2023WLD-Reg: A Data-dependent Within-layer Diversity Regularizercited by 4
2023DECOR-NET: A COVID-19 Lung Infection Segmentation Network Improved by Emphasizing Low-level Features and Decorrelating Featurescited by 4
2021The Impact of Activation Sparsity on Overfitting in Convolutional Neural Networkscited by 3
2022Concept Evolution in Deep Learning Training: A Unified Interpretation Framework and Discoveriescited by 3
2022Few-Shot Object Detection by Knowledge Distillation Using Bag-of-Visual-Words Representationscited by 3
2022XCAT -- Lightweight Quantized Single Image Super-Resolution using Heterogeneous Group Convolutions and Cross Concatenationcited by 3
2023Look Beyond Bias with Entropic Adversarial Data Augmentationcited by 3
2023Quaternion Convolutional Neural Networks: Current Advances and Future Directionscited by 3
2025Generalized Category Discovery via Token Manifold Capacity Learningcited by 3

50 most cited of 94 All 94 (JSON)