mlsanityData to 7 Oct 2026

Learning distinct features helps, provably

arXiv 2106.06012 ↗First posted 10 Jun 2021cs.LG cs.CVcited by 4

Not on the map. Why?

Abstract (arXiv)

We study the diversity of the features learned by a two-layer neural network trained with the least squares loss. We measure the diversity by the average L2-distance between the hidden-layer features and theoretically investigate how learning non-redundant distinct features affects the performance of the network. To do so, we derive novel generalization bounds depending on feature diversity based on Rademacher complexity for such networks. Our analysis proves that more distinct features at the network's units within the hidden layer lead to better generalization. We also show how to extend our results to deeper networks and different losses.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 34 of its 65 references that are in the map. First posted on arXiv 10 Jun 2021.

Builds on

34 of 65 references.

References · 34

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  3. 2016Densely Connected Convolutional Networkscited by 6,691
  4. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  5. 2016Wide Residual Networkscited by 3,057
  6. 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,540
  7. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,404
  8. 2017A Closer Look at Memorization in Deep Networkscited by 936
  9. 2017Exploring Generalization in Deep Learningcited by 595
  10. 2017Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Datacited by 527
  11. 2019Fantastic Generalization Measures and Where to Find Themcited by 507
  12. 2019Implicit Regularization in Deep Matrix Factorizationcited by 402
22 more
  1. 2017Size-Independent Sample Complexity of Neural Networkscited by 289
  2. 2018Deep Anomaly Detection Using Geometric Transformationscited by 269
  3. 2019Uniform convergence may be unable to explain generalization in deep learningcited by 223
  4. 2020Few-Shot Learning via Learning the Representation, Provablycited by 213
  5. 2019SGD on Neural Networks Learns Functions of Increasing Complexitycited by 203
  6. 2015The Benefit of Multitask Representation Learningcited by 194
  7. 2021A Universal Law of Robustness via Isoperimetrycited by 150
  8. 2017Generalization in Deep Learningcited by 146
  9. 2018Towards Understanding the Role of Over-Parametrization in Generalization of Neural Networkscited by 140
  10. 2015Reducing Overfitting in Deep Networks by Decorrelating Representationscited by 94
  11. 2017Regularization for Deep Learning: A Taxonomycited by 91
  12. 2020Do Wider Neural Networks Really Help Adversarial Robustness?cited by 79
  13. 2020Why Do Better Loss Functions Lead to Less Transferable Features?cited by 73
  14. 2021Tighter expected generalization error bounds via Wasserstein distancecited by 43
  15. 2020Provable Representation Learning for Imitation Learning via Bi-level Optimizationcited by 41
  16. 2016Diverse Neural Network Learns True Target Functionscited by 33
  17. 2019Theoretical evidence for adversarial robustness through randomizationcited by 33
  18. 2017Theory of Deep Learning III: explaining the non-overfitting puzzlecited by 32
  19. 2019Gradient Descent Finds Global Minima for Generalizable Deep Neural Networks of Practical Sizescited by 29
  20. 2018Depth with Nonlinearity Creates No Bad Local Minima in ResNetscited by 17
  21. 2016Improved Dropout for Shallow and Deep Learningcited by 11
  22. 2015On the Generalization Error Bounds of Neural Networks under Diversity-Inducing Mutual Angular Regularizationcited by 4

Built on by

Cited by year

Citing papers per year: 2023: 2; 2024: 2.

4 papers.

  1. 2024WARM: On the Benefits of Weight Averaged Reward Modelscited by 145
  2. 2023WLD-Reg: A Data-dependent Within-layer Diversity Regularizercited by 4
  3. 2024Channel-wise Feature Decorrelation for Enhanced Learned Image Compressioncited by 2
  4. 2023Convolutional autoencoder-based multimodal one-class classificationcited by 1