mlsanityData to 14 Oct 2026

Class-wise Activation Unravelling the Engima of Deep Double Descent

arXiv 2405.07679 ↗First posted 13 May 2024cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Double descent presents a counter-intuitive aspect within the machine learning domain, and researchers have observed its manifestation in various models and tasks. While some theoretical explanations have been proposed for this phenomenon in specific contexts, an accepted theory for its occurring mechanism in deep learning remains yet to be established. In this study, we revisited the phenomenon of double descent and discussed the conditions of its occurrence. This paper introduces the concept of class-activation matrices and a methodology for estimating the effective complexity of functions, on which we unveil that over-parameterized models exhibit more distinct and simpler class patterns in hidden activations compared to under-parameterized ones. We further looked into the interpolation of noisy labelled data among clean representations and demonstrated overfitting w.r.t. expressive capacity. By comprehensively analysing hypotheses and presenting corresponding empirical evidence that either validates or contradicts these hypotheses, we aim to provide fresh insights into the phenomenon of double descent and benign over-parameterization and facilitate future explorations. By comprehensively studying different hypotheses and the corresponding empirical evidence either supports or challenges these hypotheses, our goal is to offer new insights into the phenomena of double descent and benign over-parameterization, thereby enabling further explorations in the field. The source code is available at https://github.com/Yufei-Gu-451/sparse-generalization.git.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 34 of its 50 references that are in the map. First posted on arXiv 13 May 2024.

Builds on

34 of 50 references.

References · 34

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2020Scaling Laws for Neural Language Modelscited by 6,586
  4. 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 927
  5. 2015Deep Learning and the Information Bottleneck Principlecited by 802
  6. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 787
  7. 2019Deep Double Descent: Where Bigger Models and More Data Hurtcited by 672
  8. 2019Benign Overfitting in Linear Regressioncited by 549
  9. 2016Learning Structured Sparsity in Deep Neural Networkscited by 541
  10. 2019Surprises in High-Dimensional Ridgeless Least Squares Interpolationcited by 513
  11. 2019The generalization error of random features regression: Precise asymptotics and double descent curvecited by 413
  12. 2019Confident Learning: Estimating Uncertainty in Dataset Labelscited by 413
22 more
  1. 2021Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarkscited by 395
  2. 2016On the Expressive Power of Deep Neural Networkscited by 306
  3. 2017High-dimensional dynamics of generalization error in neural networkscited by 241
  4. 2019Two models of double descent for weak featurescited by 195
  5. 2019An Improved Analysis of Training Over-parameterized Deep Neural Networkscited by 121
  6. 2020Rethinking Bias-Variance Trade-off for Generalization of Neural Networkscited by 107
  7. 2020Classification vs regression in overparameterized regimes: Does the loss function matter?cited by 97
  8. 2022Robust Training under Label Noise by Over-parameterizationcited by 93
  9. 2019A Model of Double Descent for High-dimensional Binary Linear Classificationcited by 81
  10. 2021Model Complexity of Deep Learning: A Surveycited by 65
  11. 2021A Farewell to the Bias-Variance Tradeoff? An Overview of the Theory of Overparameterized Machine Learningcited by 57
  12. 2022Can Neural Nets Learn the Same Model Twice? Investigating Reproducibility and Double Descent from the Decision Boundary Perspectivecited by 57
  13. 2020A Brief Prehistory of Double Descentcited by 44
  14. 2021Benign Overfitting in Multiclass Classification: All Roads Lead to Interpolationcited by 42
  15. 2020Provable Benefits of Overparameterization in Model Compression: From Double Descent to Pruning Neural Networkscited by 40
  16. 2017Theory of Deep Learning III: explaining the non-overfitting puzzlecited by 32
  17. 2022Sparse Double Descent: Where Network Pruning Aggravates Overfittingcited by 26
  18. 2021Provable Generalization of SGD-trained Neural Networks of Any Width in the Presence of Adversarial Label Noisecited by 17
  19. 2023Benign Overfitting in Deep Neural Networks under Lazy Trainingcited by 10
  20. 2022Deep Double Descent via Smooth Interpolationcited by 8
  21. 2023Unraveling the Enigma of Double Descent: An In-depth Analysis through the Lens of Learned Feature Spacecited by 6
  22. 2022Investigating the Impact of Model Width and Density on Generalization in Presence of Label Noisecited by 3

Built on by

Not cited yet.