mlsanityData to 7 Oct 2026

Trap of Feature Diversity in the Learning of MLPs

arXiv 2112.00980 ↗First posted 2 Dec 2021cs.LG cs.AIcited by 4

Not on the map. Why?

Abstract (arXiv)

In this paper, we focus on a typical two-phase phenomenon in the learning of multi-layer perceptrons (MLPs), and we aim to explain the reason for the decrease of feature diversity in the first phase. Specifically, people find that, in the training of MLPs, the training loss does not decrease significantly until the second phase. To this end, we further explore the reason why the diversity of features over different samples keeps decreasing in the first phase, which hurts the optimization of MLPs. We explain such a phenomenon in terms of the learning dynamics of MLPs. Furthermore, we theoretically explain why four typical operations can alleviate the decrease of the feature diversity.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 42 of its 55 references that are in the map. First posted on arXiv 2 Dec 2021.

Builds on

42 of 55 references.

References · 42

  1. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  2. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  3. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  4. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
  5. 2018Neural Network Acceptability Judgmentscited by 952
  6. 2017A Closer Look at Memorization in Deep Networkscited by 936
  7. 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 748
  8. 2019Deep Double Descent: Where Bigger Models and More Data Hurtcited by 667
  9. 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 655
  10. 2017The Implicit Bias of Gradient Descent on Separable Datacited by 649
  11. 2017Deep Neural Networks as Gaussian Processescited by 617
  12. 2017Opening the Black Box of Deep Neural Networks via Informationcited by 494
30 more
  1. 2017Information-theoretic analysis of generalization capability of learning algorithmscited by 299
  2. 2018On the Optimization of Deep Networks: Implicit Acceleration by Overparameterizationcited by 280
  3. 2014The Loss Surfaces of Multilayer Networkscited by 278
  4. 2017Train longer, generalize better: closing the generalization gap in large batch training of neural networkscited by 278
  5. 2017Three Factors Influencing Minima in SGDcited by 270
  6. 2020Neural Architecture Search without Trainingcited by 241
  7. 2017High-dimensional dynamics of generalization error in neural networkscited by 238
  8. 2016Deep Learning without Poor Local Minimacited by 233
  9. 2019A Tail-Index Analysis of Stochastic Gradient Noise in Deep Neural Networkscited by 213
  10. 2018Sensitivity and Generalization in Neural Networks: an Empirical Studycited by 187
  11. 2018Evaluating the Robustness of Neural Networks: An Extreme Value Theory Approachcited by 181
  12. 2016Toward Deeper Understanding of Neural Networks: The Power of Initialization and a Dual View on Expressivitycited by 138
  13. 2017Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practicecited by 132
  14. 2016Information Dropout: Learning Optimal Representations Through Noisy Computationcited by 131
  15. 2018Convergence of Gradient Descent on Separable Datacited by 108
  16. 2016Identity Matters in Deep Learningcited by 107
  17. 2020Rethinking Bias-Variance Trade-off for Generalization of Neural Networkscited by 107
  18. 2017Spurious Local Minima are Common in Two-Layer ReLU Neural Networkscited by 104
  19. 2019Traditional and Heavy-Tailed Self Regularization in Neural Network Modelscited by 104
  20. 2020The Neural Tangent Kernel in High Dimensions: Triple Descent and a Multi-Scale Theory of Generalizationcited by 101
  21. 2020Double Trouble in Double Descent : Bias and Variance(s) in the Lazy Regimecited by 95
  22. 2018An analytic theory of generalization dynamics and transfer learning in deep linear networkscited by 84
  23. 2020Triple descent and the two kinds of overfitting: Where & why do they appear?cited by 64
  24. 2019Generalization bounds for deep convolutional neural networkscited by 58
  25. 2020Implicit Regularization of Random Feature Modelscited by 54
  26. 2019Stiffness: A New Perspective on Generalization in Neural Networkscited by 46
  27. 2018Understanding training and generalization in deep learning by Fourier analysiscited by 43
  28. 2016Regularizing CNNs with Locally Constrained Decorrelationscited by 41
  29. 2020Early Stopping in Deep Networks: Double Descent and How to Eliminate itcited by 40
  30. 2018On Tighter Generalization Bound for Deep Neural Networks: CNNs, ResNets, and Beyondcited by 24

Built on by

Cited by year

Citing papers per year: 2022: 2; 2023: 1; 2024: 1.

4 papers.

  1. 2024Tracking the Feature Dynamics in LLM Training: A Mechanistic Studycited by 15
  2. 2022Batch Normalization Is Blind to the First and Second Derivatives of the Losscited by 3
  3. 2022Towards Theoretical Analysis of Transformation Complexity of ReLU DNNscited by 2
  4. 2023Unified Batch Normalization: Identifying and Alleviating the Feature Condensation in Batch Normalization and a Unified Frameworkcited by 0