mlsanityData to 7 Oct 2026

Deep Learning is Singular, and That's Good

arXiv 2010.11560 ↗First posted 22 Oct 2020cs.LGcited by 49

Not on the map. Why?

Abstract (arXiv)

In singular models, the optimal set of parameters forms an analytic set with singularities and classical statistical inference cannot be applied to such models. This is significant for deep learning as neural networks are singular and thus "dividing" by the determinant of the Hessian or employing the Laplace approximation are not appropriate. Despite its potential for addressing fundamental issues in deep learning, singular learning theory appears to have made little inroads into the developing canon of deep learning theory. Via a mix of theory and experiment, we present an invitation to singular learning theory as a vehicle for understanding deep learning and suggest important future work to make singular learning theory directly applicable to how deep learning is performed in practice.

Key facts. Cited by 49 papers in the map (data through 7 Oct 2026). First posted on arXiv 22 Oct 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 3; 2022: 1; 2023: 9; 2024: 6; 2025: 13; 2026: 17.

49 papers.

Sort:

By area:

  1. 2023The Local Learning Coefficient: A Singularity-Aware Complexity Measurecited by 50
  2. 2024Loss Landscape Degeneracy and Stagewise Development in Transformerscited by 39
  3. 2023Dynamical versus Bayesian Phase Transitions in a Toy Model of Superpositioncited by 28
  4. 2021What can linear interpolation of neural network loss landscapes tell us?cited by 27
  5. 2025Hidden Breakthroughs in Language Model Trainingcited by 15
  6. 2024Using Degeneracy in the Loss Landscape for Mechanistic Interpretabilitycited by 12
  7. 2025Dynamics of Transient Structure in In-Context Linear Regression Transformerscited by 12
  8. 2025Bayesian Influence Functions for Hessian-Free Data Attributioncited by 12
  9. 2023The Interpolating Information Criterion for Overparameterized Modelscited by 11
  10. 2025Learning on a Razor's Edge: Identifiability and Singularity of Polynomial Neural Networkscited by 9
  11. 2025Physics of Skill Learningcited by 8
  12. 2024Estimating the Local Learning Coefficient at Scalecited by 6
37 more
  1. 2026Patterning: The Dual of Interpretabilitycited by 6
  2. 2026Topology and Geometry of the Learning Space of ReLU Networks: Connectivity and Singularitiescited by 6
  3. 2021Multirate Training of Neural Networkscited by 5
  4. 2023Bayesian Free Energy of Deep ReLU Neural Network in Overparametrized Casescited by 5
  5. 2023Functional Equivalence and Path Connectivity of Reducible Hyperbolic Tangent Networkscited by 5
  6. 2025Identifying Sparsely Active Circuits Through Local Loss Landscape Decompositioncited by 5
  7. 2025From Global to Local: A Scalable Benchmark for Local Posterior Samplingcited by 5
  8. 2022Recent Advances in Algebraic Geometry and Bayesian Statisticscited by 4
  9. 2023Free energy of Bayesian Convolutional Neural Network with Skip Connectioncited by 4
  10. 2023A PAC-Bayesian Perspective on the Interpolating Information Criterioncited by 4
  11. 2025Connecting Independently Trained Modes via Layer-Wise Connectivitycited by 3
  12. 2026Stagewise Reinforcement Learning and the Geometry of the Regret Landscapecited by 3
  13. 2024Review and Prospect of Algebraic Research in Equivalent Framework between Statistical Mechanics and Machine Learning Theorycited by 2
  14. 2025Almost Bayesian: The Fractal Dynamics of Stochastic Gradient Descentcited by 2
  15. 2023Variational Bayesian Neural Networks via Resolution of Singularitiescited by 1
  16. 2024Upper Bound of Bayesian Generalization Error in Partial Concept Bottleneck Model (CBM): Partial CBM outperforms naive CBMcited by 1
  17. 2024Temperature Optimization for Bayesian Deep Learningcited by 1
  18. 2025Emergence of Computational Structure in a Neural Network Physics Simulatorcited by 1
  19. 2026Grokking From Abstraction to Intelligencecited by 1
  20. 2026Mechanistic Anomaly Detection via Functional Attributioncited by 1
  21. 2026The Normalized Maximum Likelihood for Regular Non-Smooth Models: Measure-Theoretic Foundations and Geometric Samplingcited by 1
  22. 2021Geometry of Program Synthesiscited by 0
  23. 2023Proximity to Losslessly Compressible Parameterscited by 0
  24. 2025Non-identifiability distinguishes Neural Networks among Parametric Modelscited by 0
  25. 2025Using physics-inspired Singular Learning Theory to understand grokking & other phase transitions in modern neural networkscited by 0
  26. 2025An Equivariance Toolbox for Learning Dynamicscited by 0
  27. 2026A Regularization-Sharpness Tradeoff for Linear Interpolatorscited by 0
  28. 2026The Geometric Cost of Normalization: Affine Bounds on the Bayesian Complexity of Neural Networkscited by 0
  29. 2026Noise-Debiased Thermodynamic Variance for Local Learning Coefficient Probescited by 0
  30. 2026Singular Learning and Occam's Razor in Deep Monomial Networkscited by 0
  31. 2026A Statistical Difference between Single-Layer Learning and Hierarchical Learning in Wide Neural Networkscited by 0
  32. 2026Langevin dynamics along the zero set of real-analytic potentialscited by 0
  33. 2026Wrong Operator or Blind Design? A Reference-Free Diagnostic for Physics-Informed Coefficient Learningcited by 0
  34. 2026Exact Algebraic Computation of Learning Coefficients for Two-Dimensional Singular Modelscited by 0
  35. 2026Percolation Dynamics in Optimization: Variance Cascades and Nested Symmetrycited by 0
  36. 2026Symmetries and Singularitiescited by 0
  37. 2026Misalignment of Low-Loss Regions Causes Grokkingcited by 0