mlsanityData to 7 Oct 2026

Intraclass clustering: an implicit learning ability that regularizes DNNs

arXiv 2103.06733 ↗First posted 11 Mar 2021cs.LG cs.AI cs.CVcited by 6

Not on the map. Why?

Abstract (arXiv)

Several works have shown that the regularization mechanisms underlying deep neural networks' generalization performances are still poorly understood. In this paper, we hypothesize that deep neural networks are regularized through their ability to extract meaningful clusters among the samples of a class. This constitutes an implicit form of regularization, as no explicit training mechanisms or supervision target such behaviour. To support our hypothesis, we design four different measures of intraclass clustering, based on the neuron- and layer-level representations of the training data. We then show that these measures constitute accurate predictors of generalization performance across variations of a large set of hyperparameters (learning rate, batch size, optimizer, weight decay, dropout rate, data augmentation, network depth and width).

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 32 of its 39 references that are in the map. First posted on arXiv 11 Mar 2021.

Builds on

32 of 39 references.

References · 32

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  4. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  5. 2016Wide Residual Networkscited by 3,057
  6. 2013Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Mapscited by 2,348
  7. 2013Visualizing and Understanding Convolutional Networkscited by 2,278
  8. 2016Understanding deep learning requires rethinking generalizationcited by 1,657
  9. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
  10. 2019When Does Label Smoothing Help?cited by 985
  11. 2017A Closer Look at Memorization in Deep Networkscited by 936
  12. 2017Network Dissection: Quantifying Interpretability of Deep Visual Representationscited by 703
20 more
  1. 2019Linear Mode Connectivity and the Lottery Ticket Hypothesiscited by 627
  2. 2019Fantastic Generalization Measures and Where to Find Themcited by 507
  3. 2014In Search of the Real Inductive Bias: On the Role of Implicit Regularization in Deep Learningcited by 339
  4. 2015Understanding Neural Networks Through Deep Visualizationcited by 332
  5. 2017The Marginal Value of Adaptive Gradient Methods in Machine Learningcited by 248
  6. 2018Gradient Descent Happens in a Tiny Subspacecited by 208
  7. 2016What makes ImageNet good for transfer learning?cited by 192
  8. 2018Sensitivity and Generalization in Neural Networks: an Empirical Studycited by 187
  9. 2020Deep learning versus kernel learning: an empirical study of loss landscape geometry and the time evolution of the Neural Tangent Kernelcited by 185
  10. 2014Object Detectors Emerge in Deep Scene CNNscited by 178
  11. 2020The Break-Even Point on Optimization Trajectories of Deep Neural Networkscited by 149
  12. 2017L2 Regularization versus Batch and Weight Normalizationcited by 121
  13. 2020The Early Phase of Neural Network Trainingcited by 114
  14. 2018Three Mechanisms of Weight Decay Regularizationcited by 112
  15. 2018On the importance of single directions for generalizationcited by 99
  16. 2017Critical Learning Periods in Deep Neural Networkscited by 78
  17. 2019Time Matters in Regularizing Deep Networks: Weight Decay and Data Augmentation Affect Early Learning Dynamics, Matter Little Near Convergencecited by 54
  18. 2019Stiffness: A New Perspective on Generalization in Neural Networkscited by 46
  19. 2018Semi-Supervised Learning via Compact Latent Space Clusteringcited by 15
  20. 2020Selectivity considered harmful: evaluating the causal impact of class selectivity in DNNscited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 1; 2024: 0; 2025: 2; 2026: 2.

6 papers.

  1. 2023Gaussian Latent Representations for Uncertainty Estimation using Mahalanobis Distance in Deep Classifierscited by 10
  2. 2026Explainable AI in Speaker Recognition -- Making Latent Representations Understandablecited by 3
  3. 2025The Advantage of Fine-Grained Trainingcited by 1
  4. 2025HOLE: Homological Observation of Latent Embeddings for Neural Network Interpretabilitycited by 1
  5. 2022Towards understanding deep learning with the natural clustering priorcited by 0
  6. 2026Interpreting hierarchical organisation of speaker embeddingscited by 0