mlsanityData to 14 Oct 2026

Representation Gap: Explaining the Unreasonable Effectiveness of Neural Networks from a Geometric Perspective

arXiv 2605.21692 ↗First posted 20 May 2026cs.LG stat.MLcited by 0

Not on the map. Why?

Abstract (arXiv)

Characterizing precisely the asymptotic generalization error of neural networks using parameters that can be estimated efficiently is a crucial problem in machine learning, which relies heavily on heuristics and practitioners' intuition to make key design choices. In order to mitigate this issue, we introduce the Representation Gap, a metric closely related to the generalization error, but admitting better-behaved asymptotic dynamics. Focusing on equivariant diffusion models and leveraging results from optimal quantization and point-process theory, we derive a precise asymptotic equivalent of the Representation Gap and show that it is governed by a single parameter, the textitintrinsic dimension of the task, which is easy to interpret, efficient to estimate, and can be linked to the equivariances of common neural network architectures. We show that this asymptotic dynamic also extends to a broader range of tasks and training algorithms. Finally, we demonstrate empirically that our asymptotic law and intrinsic dimension estimation are accurate on a wide range of synthetic datasets, where these quantities are known, as well as on more realistic datasets, where we obtain results consistent with the related literature.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 33 of its 73 references that are in the map. First posted on arXiv 20 May 2026.

Builds on

33 of 73 references.

References · 33

  1. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  3. 2020Denoising Diffusion Implicit Modelscited by 9,781
  4. 2020Scaling Laws for Neural Language Modelscited by 6,586
  5. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,101
  6. 2015Variational Inference with Normalizing Flowscited by 2,170
  7. 2016Group Equivariant Convolutional Networkscited by 1,165
  8. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 787
  9. 2022Equivariant Diffusion for Molecule Generation in 3Dcited by 677
  10. 2019PointFlow: 3D Point Cloud Generation with Continuous Normalizing Flowscited by 488
  11. 2021The Intrinsic Dimension of Images and Its Impact on Learningcited by 315
  12. 2015A note on the evaluation of generative modelscited by 292
21 more
  1. 2019Intrinsic dimension of data representations in deep neural networkscited by 282
  2. 2021Fit without fear: remarkable mathematical phenomena of deep learning through the prism of interpolationcited by 132
  3. 2024An analytic theory of creativity in convolutional diffusion modelscited by 104
  4. 2021Provably Strict Generalisation Benefit for Equivariant Modelscited by 91
  5. 2024Advances in 3D Generation: A Surveycited by 63
  6. 2018On the Intrinsic Dimensionality of Image Representationscited by 57
  7. 2024Deep Generative Models through the Lens of the Manifold Hypothesis: A Survey and New Connectionscited by 47
  8. 2024Scaling Laws For Diffusion Transformerscited by 44
  9. 2024Neural Redshift: Random Networks are not Random Functionscited by 43
  10. 2024On the Scalability of Diffusion-based Text-to-Image Generationcited by 36
  11. 2019Improved Generalization Bounds of Group Invariant / Equivariant Deep Networks via Quotient Feature Spacescited by 34
  12. 2026There Will Be a Scientific Theory of Deep Learningcited by 29
  13. 2023The Exact Sample Complexity Gain from Invariances for Kernel Regressioncited by 28
  14. 2024Bigger is not Always Better: Scaling Properties of Latent Diffusion Modelscited by 21
  15. 2019Implicit Regularization in Over-parameterized Neural Networkscited by 10
  16. 2025Origins of Creativity in Attention-Based Diffusion Modelscited by 4
  17. 2025Staying on the Manifold: Geometry-Aware Noise Injectioncited by 4
  18. 2021A manifold learning perspective on representation learning: Learning decoder and representations without an encodercited by 3
  19. 2021Equivariance and generalization in neural networkscited by 3
  20. 2024A Theoretical Study of Neural Network Expressive Power via Manifold Topologycited by 2
  21. 2018Volume of small balls and sub-Riemannian curvature in 3D contact manifoldscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Local Intrinsic Dimension of Representations Predicts Alignment and Generalization in AI Models and Human Brain
  2. 2024A Survey on Statistical Theory of Deep Learning: Approximation, Training Dynamics, and Generative Models
  3. 2025Optimal generalisation and learning transition in extensive-width shallow neural networks near interpolation
  4. 2026Geometry-aware similarity metrics for neural representations on Riemannian and statistical manifolds
  5. 2026The Model Shape Behind Scaling Laws: A Gradient Superposition Perspective
  6. 2024Collective variables of neural networks: empirical time evolution and scaling laws
  7. 2020Spectral Bias and Task-Model Alignment Explain Generalization in Kernel Regression and Infinitely Wide Neural Networks
  8. 2026The Predictive-Causal Gap: An Impossibility Theorem and Large-Scale Neural Evidence