mlsanityData to 7 Oct 2026

WLD-Reg: A Data-dependent Within-layer Diversity Regularizer

arXiv 2301.01352 ↗First posted 3 Jan 2023cs.LG cs.CVcited by 4

Not on the map. Why?

Abstract (arXiv)

Neural networks are composed of multiple layers arranged in a hierarchical structure jointly trained with a gradient-based optimization, where the errors are back-propagated from the last layer back to the first one. At each optimization step, neurons at a given layer receive feedback from neurons belonging to higher layers of the hierarchy. In this paper, we propose to complement this traditional 'between-layer' feedback with additional 'within-layer' feedback to encourage the diversity of the activations within the same layer. To this end, we measure the pairwise similarity between the outputs of the neurons and use it to model the layer's overall diversity. We present an extensive empirical study confirming that the proposed approach enhances the performance of several state-of-the-art neural network models in multiple tasks. The code is publically available at urlhttps://github.com/firasl/AAAI-23-WLD-Reg

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 38 of its 64 references that are in the map. First posted on arXiv 3 Jan 2023.

Builds on

38 of 64 references.

References · 38

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  3. 2016Densely Connected Convolutional Networkscited by 6,691
  4. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  5. 2016Wide Residual Networkscited by 3,057
  6. 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,540
  7. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,818
  8. 2021MLP-Mixer: An all-MLP Architecture for Visioncited by 1,441
  9. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,404
  10. 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 919
  11. 2019Deep Double Descent: Where Bigger Models and More Data Hurtcited by 667
  12. 2017Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Datacited by 527
26 more
  1. 2019Implicit Regularization in Deep Matrix Factorizationcited by 402
  2. 2021Pay Attention to MLPscited by 367
  3. 2018Filter Pruning via Geometric Median for Deep Convolutional Neural Networks Accelerationcited by 365
  4. 2021FNet: Mixing Tokens with Fourier Transformscited by 343
  5. 2018Deep Anomaly Detection Using Geometric Transformationscited by 269
  6. 2017High-dimensional dynamics of generalization error in neural networkscited by 238
  7. 2019Uniform convergence may be unable to explain generalization in deep learningcited by 223
  8. 2018Towards Understanding the Role of Over-Parametrization in Generalization of Neural Networkscited by 140
  9. 2016Energy-based Generative Adversarial Networkcited by 119
  10. 2015Reducing Overfitting in Deep Networks by Decorrelating Representationscited by 94
  11. 2017Regularization for Deep Learning: A Taxonomycited by 91
  12. 2019Batch Active Learning Using Determinantal Point Processescited by 42
  13. 2016Regularizing CNNs with Locally Constrained Decorrelationscited by 41
  14. 2016Diverse Neural Network Learns True Target Functionscited by 33
  15. 2017Theory of Deep Learning III: explaining the non-overfitting puzzlecited by 32
  16. 2018Leveraging Filter Correlations for Deep Model Compressioncited by 23
  17. 2019Balanced Ranking with Diversity Constraintscited by 20
  18. 2019Exact sampling of determinantal point processes with sublinear time preprocessingcited by 19
  19. 2020Learning Unbiased Representations via R'enyi Minimizationcited by 18
  20. 2020Spatial Contrastive Learning for Few-Shot Classificationcited by 17
  21. 2019Learning Nonsymmetric Determinantal Point Processescited by 15
  22. 2016Improved Dropout for Shallow and Deep Learningcited by 11
  23. 2017Diversity-Promoting Bayesian Learning of Latent Variable Modelscited by 5
  24. 2022Reducing Redundancy in the Bottleneck Representation of the Autoencoderscited by 5
  25. 2015On the Generalization Error Bounds of Neural Networks under Diversity-Inducing Mutual Angular Regularizationcited by 4
  26. 2021Learning distinct features helps, provablycited by 4

Built on by

Cited by year

Citing papers per year: 2023: 3; 2024: 1.

4 papers.

  1. 2023Variance-Covariance Regularization Improves Representation Learningcited by 13
  2. 2024Channel-wise Feature Decorrelation for Enhanced Learned Image Compressioncited by 2
  3. 2023Convolutional autoencoder-based multimodal one-class classificationcited by 1
  4. 2023On Feature Diversity in Energy-based Modelscited by 0