# WLD-Reg: A Data-dependent Within-layer Diversity Regularizer

arXiv: https://arxiv.org/abs/2301.01352 · HTML: https://mlsanity.com/paper/2301.01352/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 38 of its 64 references that are in the map.
- First posted on arXiv 3 Jan 2023.

## Builds on (most cited first, 30 of 38)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [Wide Residual Networks](https://mlsanity.com/paper/1605.07146/) (2016)
- [Aggregated Residual Transformations for Deep Neural Networks](https://mlsanity.com/paper/1611.05431/) (2016)
- [Barlow Twins: Self-Supervised Learning via Redundancy Reduction](https://mlsanity.com/paper/2103.03230/) (2021)
- [MLP-Mixer: An all-MLP Architecture for Vision](https://mlsanity.com/paper/2105.01601/) (2021)
- [Sharpness-Aware Minimization for Efficiently Improving Generalization](https://mlsanity.com/paper/2010.01412/) (2020)
- [Reconciling modern machine learning practice and the bias-variance trade-off](https://mlsanity.com/paper/1812.11118/) (2018)
- [Deep Double Descent: Where Bigger Models and More Data Hurt](https://mlsanity.com/paper/1912.02292/) (2019)
- [Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Data](https://mlsanity.com/paper/1703.11008/) (2017)
- [Implicit Regularization in Deep Matrix Factorization](https://mlsanity.com/paper/1905.13655/) (2019)
- [Pay Attention to MLPs](https://mlsanity.com/paper/2105.08050/) (2021)
- [Filter Pruning via Geometric Median for Deep Convolutional Neural Networks Acceleration](https://mlsanity.com/paper/1811.00250/) (2018)
- [FNet: Mixing Tokens with Fourier Transforms](https://mlsanity.com/paper/2105.03824/) (2021)
- [Deep Anomaly Detection Using Geometric Transformations](https://mlsanity.com/paper/1805.10917/) (2018)
- [High-dimensional dynamics of generalization error in neural networks](https://mlsanity.com/paper/1710.03667/) (2017)
- [Uniform convergence may be unable to explain generalization in deep learning](https://mlsanity.com/paper/1902.04742/) (2019)
- [Towards Understanding the Role of Over-Parametrization in Generalization of Neural Networks](https://mlsanity.com/paper/1805.12076/) (2018)
- [Energy-based Generative Adversarial Network](https://mlsanity.com/paper/1609.03126/) (2016)
- [Reducing Overfitting in Deep Networks by Decorrelating Representations](https://mlsanity.com/paper/1511.06068/) (2015)
- [Regularization for Deep Learning: A Taxonomy](https://mlsanity.com/paper/1710.10686/) (2017)
- [Batch Active Learning Using Determinantal Point Processes](https://mlsanity.com/paper/1906.07975/) (2019)
- [Regularizing CNNs with Locally Constrained Decorrelations](https://mlsanity.com/paper/1611.01967/) (2016)
- [Diverse Neural Network Learns True Target Functions](https://mlsanity.com/paper/1611.03131/) (2016)
- [Theory of Deep Learning III: explaining the non-overfitting puzzle](https://mlsanity.com/paper/1801.00173/) (2017)
- [Leveraging Filter Correlations for Deep Model Compression](https://mlsanity.com/paper/1811.10559/) (2018)
- [Balanced Ranking with Diversity Constraints](https://mlsanity.com/paper/1906.01747/) (2019)
- [Exact sampling of determinantal point processes with sublinear time preprocessing](https://mlsanity.com/paper/1905.13476/) (2019)

## Built on by (most cited first, 4 of 4)

- [Variance-Covariance Regularization Improves Representation Learning](https://mlsanity.com/paper/2306.13292/) (2023)
- [Channel-wise Feature Decorrelation for Enhanced Learned Image Compression](https://mlsanity.com/paper/2403.10936/) (2024)
- [Convolutional autoencoder-based multimodal one-class classification](https://mlsanity.com/paper/2309.14090/) (2023)
- [On Feature Diversity in Energy-based Models](https://mlsanity.com/paper/2306.01489/) (2023)

## Abstract (arXiv)

Neural networks are composed of multiple layers arranged in a hierarchical structure jointly trained with a gradient-based optimization, where the errors are back-propagated from the last layer back to the first one. At each optimization step, neurons at a given layer receive feedback from neurons belonging to higher layers of the hierarchy. In this paper, we propose to complement this traditional 'between-layer' feedback with additional 'within-layer' feedback to encourage the diversity of the activations within the same layer. To this end, we measure the pairwise similarity between the outputs of the neurons and use it to model the layer's overall diversity. We present an extensive empirical study confirming that the proposed approach enhances the performance of several state-of-the-art neural network models in multiple tasks. The code is publically available at urlhttps://github.com/firasl/AAAI-23-WLD-Reg