mlsanityData to 7 Oct 2026

Double Descent of Discrepancy: A Task-, Data-, and Model-Agnostic Phenomenon

arXiv 2305.15907 ↗First posted 25 May 2023cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

In this paper, we studied two identically-trained neural networks (i.e. networks with the same architecture, trained on the same dataset using the same algorithm, but with different initialization) and found that their outputs discrepancy on the training dataset exhibits a "double descent" phenomenon. We demonstrated through extensive experiments across various tasks, datasets, and network architectures that this phenomenon is prevalent. Leveraging this phenomenon, we proposed a new early stopping criterion and developed a new method for data quality assessment. Our results show that a phenomenon-driven approach can benefit deep learning research both in theoretical understanding and practical applications.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 27 of its 32 references that are in the map. First posted on arXiv 25 May 2023.

Builds on

27 of 32 references.

References · 27

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  3. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,282
  4. 2016Densely Connected Convolutional Networkscited by 6,691
  5. 2017Squeeze-and-Excitation Networkscited by 4,763
  6. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  7. 2020Implicit Neural Representations with Periodic Activation Functionscited by 2,214
  8. 2018On the Spectral Bias of Neural Networkscited by 1,321
  9. 2017Deep Image Priorcited by 1,097
  10. 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 919
  11. 2019Deep Double Descent: Where Bigger Models and More Data Hurtcited by 667
  12. 2017Deep Layer Aggregationcited by 436
15 more
  1. 2019Frequency Principle: Fourier Analysis Sheds Light on Deep Neural Networkscited by 360
  2. 2018Training behavior of deep neural network in frequency domaincited by 174
  3. 2019Gradient Descent with Early Stopping is Provably Robust to Label Noise for Overparameterized Neural Networkscited by 158
  4. 2020Frequency Bias in Neural Networks for Input of Non-Uniform Densitycited by 140
  5. 2019The Convergence Rate of Neural Networks for Learned Functions of Different Frequenciescited by 134
  6. 2020Rethinking Bias-Variance Trade-off for Generalization of Neural Networkscited by 107
  7. 2020Double Trouble in Double Descent : Bias and Variance(s) in the Lazy Regimecited by 95
  8. 2017Learning with Biased Complementary Labelscited by 93
  9. 2021Early Stopping for Deep Image Priorcited by 44
  10. 2020Early Stopping in Deep Networks: Double Descent and How to Eliminate itcited by 40
  11. 2017Early Stopping without a Validation Setcited by 39
  12. 2021When and how epochwise double descent happenscited by 20
  13. 2021Channel-Wise Early Stopping without a Validation Set via NNK Polytope Interpolationcited by 11
  14. 2021Disparity Between Batches as a Signal for Early Stoppingcited by 9
  15. 2022Intersection of Parallels as an Early Stopping Criterioncited by 3

Built on by

Not cited yet.