mlsanityData to 7 Oct 2026

Deep Learning is Not So Mysterious or Different

arXiv 2503.02113 ↗First posted 3 Mar 2025cs.LG stat.MLcited by 55

Not on the map. Why?

Abstract (arXiv)

Deep neural networks are often seen as different from other model classes by defying conventional notions of generalization. Popular examples of anomalous generalization behaviour include benign overfitting, double descent, and the success of overparametrization. We argue that these phenomena are not distinct to neural networks, or particularly mysterious. Moreover, this generalization behaviour can be intuitively understood, and rigorously characterized, using long-standing generalization frameworks such as PAC-Bayes and countable hypothesis bounds. We present soft inductive biases as a key unifying principle in explaining these phenomena: rather than restricting the hypothesis space to avoid overfitting, embrace a flexible hypothesis space, with a soft preference for simpler solutions that are consistent with the data. This principle can be encoded in many model classes, and thus deep learning is not as mysterious or different from other model classes as it might seem. However, we also highlight how deep learning is relatively distinct in other ways, such as its ability for representation learning, phenomena such as mode connectivity, and its relative universality.

Key facts. Cited by 55 papers in the map (data through 7 Oct 2026). Builds on 63 of its 110 references that are in the map. First posted on arXiv 3 Mar 2025.

Builds on

63 of 110 references.

References · 63

  1. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  2. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  3. 2016Understanding deep learning requires rethinking generalizationcited by 1,657
  4. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,404
  5. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
  6. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
  7. 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,194
  8. 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 919
  9. 2019Deep Double Descent: Where Bigger Models and More Data Hurtcited by 667
  10. 2019Linear Mode Connectivity and the Lottery Ticket Hypothesiscited by 627
  11. 2017Deep Neural Networks as Gaussian Processescited by 617
  12. 2018Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNscited by 585
38 more
  1. 2019Benign Overfitting in Linear Regressioncited by 544
  2. 2017Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Datacited by 527
  3. 2019Surprises in High-Dimensional Ridgeless Least Squares Interpolationcited by 508
  4. 2019Fantastic Generalization Measures and Where to Find Themcited by 507
  5. 2023Large Language Models Are Zero-Shot Time Series Forecasterscited by 491
  6. 2020Bayesian Deep Learning and a Probabilistic Perspective of Generalizationcited by 480
  7. 2017Sharp Minima Can Generalize For Deep Netscited by 462
  8. 2022Git Re-Basin: Merging Models modulo Permutation Symmetriescited by 460
  9. 2015Deep Kernel Learningcited by 440
  10. 2025Muon is Scalable for LLM Trainingcited by 381
  11. 2018Measuring the Intrinsic Dimension of Objective Landscapescited by 364
  12. 2018Essentially No Barriers in Neural Network Energy Landscapecited by 328
  13. 2018Gaussian Process Behaviour in Wide Deep Neural Networkscited by 314
  14. 2018Stronger generalization bounds for deep nets via a compression approachcited by 306
  15. 2024Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunitiescited by 247
  16. 2021User-friendly introduction to PAC-Bayes boundscited by 234
  17. 2018To understand deep learning we need to understand kernel learningcited by 186
  18. 2020Feature Learning in Infinite-Width Neural Networkscited by 182
  19. 2024SOAP: Improving and Stabilizing Shampoo using Adamcited by 179
  20. 2015High-Dimensional Asymptotics of Prediction: Ridge Regression and Classificationcited by 146
  21. 2017A Bayesian Perspective on Generalization and Stochastic Gradient Descentcited by 134
  22. 2013Gaussian Process Kernels for Pattern Discovery and Extrapolationcited by 131
  23. 2019A Primer on PAC-Bayesian Learningcited by 130
  24. 2020Bridging Mode Connectivity in Loss Landscapes and Adversarial Robustnesscited by 130
  25. 2018Non-Vacuous Generalization Bounds at the ImageNet Scale: A PAC-Bayesian Compression Approachcited by 125
  26. 2020Tighter risk certificates for neural networkscited by 112
  27. 2016Topology and Geometry of Half-Rectified Network Optimizationcited by 102
  28. 2022Benign Overfitting in Two-layer Convolutional Neural Networkscited by 90
  29. 2016PAC-Bayesian Theory Meets Bayesian Inferencecited by 87
  30. 2021Loss Surface Simplexes for Mode Connecting Volumes and Fast Ensemblingcited by 75
  31. 2022PAC-Bayes Compression Bounds So Tight That They Can Explain Generalizationcited by 75
  32. 2021Residual Pathway Priors for Soft Equivariance Constraintscited by 74
  33. 2019Explaining Landscape Connectivity of Low-cost Solutions for Multilayer Netscited by 63
  34. 2021Stochastic Training is Not Necessary for Generalizationcited by 63
  35. 2023The No Free Lunch Theorem, Kolmogorov Complexity, and the Role of Inductive Biases in Machine Learningcited by 63
  36. 2022Bayesian Model Selection, the Marginal Likelihood, and Generalizationcited by 56
  37. 2019Understanding Generalization through Visualizationscited by 54
  38. 2024Scaling Laws in Linear Regression: Compute, Parameters, and Datacited by 54

50 most cited of 63 All 63 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 23; 2026: 32.

55 papers.

Sort:

By area:

  1. 2026There Will Be a Scientific Theory of Deep Learningcited by 21
  2. 2025Block-Recurrent Dynamics in Vision Transformerscited by 16
  3. 2025Local Mixtures of Experts: Essentially Free Test-Time Training via Model Mergingcited by 9
  4. 2025Models of Heavy-Tailed Mechanistic Universalitycited by 9
  5. 2025Understanding Transformers for Time Series: Rank Structure, Flow-of-ranks, and Compressibilitycited by 8
  6. 2025The False Promise of Zero-Shot Super-Resolution in Machine-Learned Operatorscited by 8
  7. 2025Scaling can lead to compositional generalizationcited by 7
  8. 2025Visual Instruction Bottleneck Tuningcited by 5
  9. 2025FastNet: Improving the physical consistency of machine-learning weather prediction models through loss function designcited by 4
  10. 2025Sharp Minima Can Generalize: A Loss Landscape Perspective On Datacited by 3
  11. 2026WinQ: Accelerating Quantization-Aware Training of Language Models Around Saddle Pointscited by 3
  12. 2026Explorative Modeling: Unlocking a Third Pretraining Axis and End-to-End Generationcited by 3
38 more
  1. 2025Uncertainty Quantification and Data Efficiency in AI: An Information-Theoretic Perspectivecited by 2
  2. 2026Algorithmic Simplification of Neural Networks with Mosaic-of-Motifscited by 2
  3. 2026Solomonoff inductioncited by 2
  4. 2026A PAC-Bayesian approach to generalization for quantum modelscited by 2
  5. 2026Statistical Properties of Training & Generalizationcited by 2
  6. 2025Optimizing Data Augmentation through Bayesian Model Selectioncited by 1
  7. 2025DNNs, Dataset Statistics, and Correlation Functionscited by 1
  8. 2026Nonlinear multi-study sparse factor analysiscited by 1
  9. 2025Lower Bounds on the MMSE of Adversarially Inferring Sensitive Featurescited by 0
  10. 2025Overfitting has a limitation: a model-independent generalization gap bound based on R'enyi entropycited by 0
  11. 2025Knowledge distillation as a pathway toward next-generation intelligent ecohydrological modeling systemscited by 0
  12. 2025Flexible inference for animal learning rules using neural networkscited by 0
  13. 2025Response to Promises and Pitfalls of Deep Kernel Learningcited by 0
  14. 2025Soft Inductive Bias Approach via Explicit Reasoning Perspectives in Inappropriate Utterance Detection Using Large Language Modelscited by 0
  15. 2025Language models as tools for investigating the distinction between possible and impossible natural languagescited by 0
  16. 2025Large language models have learned to use languagecited by 0
  17. 2025Large language models are not about natural languagecited by 0
  18. 2025Linguists should learn to love speech-based deep learning modelscited by 0
  19. 2025Soft Geometric Inductive Bias for Object Centric Dynamicscited by 0
  20. 2026Deep Exploration of Epoch-wise Double Descent in Noisy Data: Signal Separation, Large Activation, and Benign Overfittingcited by 0
  21. 2026Are Language Models Models?cited by 0
  22. 2026Parabolic Position Encoding: Vision-Centric, Principled, Extrapolatable, Generalcited by 0
  23. 2026Beyond Learning on Molecules by Weakly Supervising on Moleculescited by 0
  24. 2026Curvature-Weighted Capacity Allocation: A Minimum Description Length Framework for Layer-Adaptive Large Language Model Optimizationcited by 0
  25. 2026A Compression Perspective on Simplicity Biascited by 0
  26. 2026Across the Levels of Analysis: Explaining Predictive Processing in Humans Requires More Than Machine-Estimated Probabilitiescited by 0
  27. 2026You Can't Fight in Here! This is BBS!cited by 0
  28. 2026The Virtue of Sparsity in Complexitycited by 0
  29. 2026Estimating Implicit Regularization in Deep Learningcited by 0
  30. 2026Not-So-Strange Love: Language Models and Generative Linguistic Theories are More Compatible than They Appearcited by 0
  31. 2026Characterizing Learning in Deep Neural Networks using Tractable Algorithmic Complexity Analysiscited by 0
  32. 2026How can embedding models bind concepts?cited by 0
  33. 2026In Defense of Information Leakage in Concept-based Modelscited by 0
  34. 2026The FIL Hypothesis: Inductive Biases Help with Kernel Engineeringcited by 0
  35. 2026Requential Coding: Pushing the Limits of Model Compression with Self-Generated Training Datacited by 0
  36. 2026Semantic Space Search Trajectory Networkscited by 0
  37. 2026Benign interpolation and Occam's razorcited by 0
  38. 2026An Investigation of the NeurIPS and ICML 2025 Position Trackscited by 0

50 most cited of 55 All 55 (JSON)