mlsanityData to 7 Oct 2026

Redundant representations help generalization in wide neural networks

arXiv 2106.03485 ↗First posted 7 Jun 2021stat.ML cs.CV cs.LGcited by 9

Not on the map. Why?

Abstract (arXiv)

Deep neural networks (DNNs) defy the classical bias-variance trade-off: adding parameters to a DNN that interpolates its training data will typically improve its generalization performance. Explaining the mechanism behind this ``benign overfitting'' in deep networks remains an outstanding challenge. Here, we study the last hidden layer representations of various state-of-the-art convolutional neural networks and find that if the last hidden representation is wide enough, its neurons tend to split into groups that carry identical information, and differ from each other only by statistically independent noise. The number of such groups increases linearly with the width of the layer, but only if the width is above a critical value. We show that redundant neurons appear only when the training process reaches interpolation and the training error is zero.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Builds on 44 of its 76 references that are in the map. First posted on arXiv 7 Jun 2021.

Builds on

44 of 76 references.

References · 44

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2016Densely Connected Convolutional Networkscited by 6,691
  3. 2020Scaling Laws for Neural Language Modelscited by 6,411
  4. 2016Wide Residual Networkscited by 3,057
  5. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  6. 2016Understanding deep learning requires rethinking generalizationcited by 1,657
  7. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,479
  8. 2017Mixed Precision Trainingcited by 1,135
  9. 2017A Closer Look at Memorization in Deep Networkscited by 936
  10. 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 919
  11. 2018On Lazy Training in Differentiable Programmingcited by 671
  12. 2019Deep Double Descent: Where Bigger Models and More Data Hurtcited by 667
32 more
  1. 2017The Implicit Bias of Gradient Descent on Separable Datacited by 649
  2. 2017Deep Learning Scaling is Predictable, Empiricallycited by 556
  3. 2019Benign Overfitting in Linear Regressioncited by 544
  4. 2020Scaling Laws for Autoregressive Generative Modelingcited by 510
  5. 2019Surprises in High-Dimensional Ridgeless Least Squares Interpolationcited by 508
  6. 2018A Mean Field View of the Landscape of Two-Layers Neural Networkscited by 463
  7. 2018On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transportcited by 422
  8. 2019The generalization error of random features regression: Precise asymptotics and double descent curvecited by 410
  9. 2019Implicit Regularization in Deep Matrix Factorizationcited by 402
  10. 2021Explaining Neural Scaling Lawscited by 353
  11. 2014In Search of the Real Inductive Bias: On the Role of Implicit Regularization in Deep Learningcited by 339
  12. 2019Intrinsic dimension of data representations in deep neural networkscited by 276
  13. 2020Implicit Bias of Gradient Descent for Wide Two-layer Neural Networks Trained with the Logistic Losscited by 267
  14. 2018Characterizing Implicit Bias in Terms of Optimization Geometrycited by 259
  15. 2018Implicit Bias of Gradient Descent on Linear Convolutional Networkscited by 251
  16. 2017High-dimensional dynamics of generalization error in neural networkscited by 238
  17. 2019A Constructive Prediction of the Generalization Error Across Scalescited by 202
  18. 2021Revisiting ResNets: Improved Training and Scaling Strategiescited by 158
  19. 2019Dynamics of stochastic gradient descent for two-layer neural networks in the teacher-student setupcited by 112
  20. 2018Mean Field Analysis of Neural Networks: A Central Limit Theoremcited by 107
  21. 2020Rethinking Bias-Variance Trade-off for Generalization of Neural Networkscited by 107
  22. 2020Double Trouble in Double Descent : Bias and Variance(s) in the Lazy Regimecited by 95
  23. 2019Scaling description of generalization with number of parameters in deep learningcited by 83
  24. 2020Understanding Double Descent Requires a Fine-Grained Bias-Variance Decompositioncited by 75
  25. 2021Classifying high-dimensional Gaussian mixtures: Where kernel methods fail and neural networks succeedcited by 67
  26. 2018A jamming transition from under- to over-parametrization affects loss landscape and generalizationcited by 65
  27. 2018A Modern Take on the Bias-Variance Tradeoff in Neural Networkscited by 64
  28. 2019Explaining Landscape Connectivity of Low-cost Solutions for Multilayer Netscited by 63
  29. 2019Bad Global Minima Exist and SGD Can Reach Themcited by 49
  30. 2017Gradient Descent Learns One-hidden-layer CNN: Don't be Afraid of Spurious Local Minimacited by 40
  31. 2019Landscape Connectivity and Dropout Stability of SGD Solutions for Over-parameterized Neural Networkscited by 35
  32. 2020What causes the test error? Going beyond bias-variance via ANOVAcited by 28

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 4; 2025: 1; 2026: 1.

9 papers.

  1. 2024A Resource Model For Neural Scaling Lawcited by 8
  2. 2024Survival of the Fittest Representation: A Case Study with Modular Additioncited by 7
  3. 2023Understanding Calibration of Deep Neural Networks for Medical Image Classificationcited by 5
  4. 2023A Neural Scaling Law from Lottery Ticket Ensemblingcited by 5
  5. 2024Fine-Grained Domain Generalization with Feature Structuralizationcited by 3
  6. 2023Look-Ahead Selective Plasticity for Continual Learning of Visual Taskscited by 2
  7. 2024Sequencing the Neurome: Towards Scalable Exact Parameter Reconstruction of Black-Box Neural Networkscited by 0
  8. 2025Redundancy as a Structural Information Principle for Learning and Generalizationcited by 0
  9. 2026How Much of a Model Do We Need? Redundancy and Slimmability in Remote Sensing Foundation Modelscited by 0