mlsanityData to 7 Oct 2026

Effects of Data Geometry in Early Deep Learning

arXiv 2301.00008 ↗First posted 29 Dec 2022cs.LG cs.AIcited by 12

Not on the map. Why?

Abstract (arXiv)

Deep neural networks can approximate functions on different types of data, from images to graphs, with varied underlying structure. This underlying structure can be viewed as the geometry of the data manifold. By extending recent advances in the theoretical understanding of neural networks, we study how a randomly initialized neural network with piece-wise linear activation splits the data manifold into regions where the neural network behaves as a linear function. We derive bounds on the density of boundary of linear regions and the distance to these boundaries on the data manifold. This leads to insights into the expressivity of randomly initialized deep neural networks on non-Euclidean data sets. We empirically corroborate our theoretical results using a toy supervised learning problem. Our experiments demonstrate that number of linear regions varies across manifolds and the results hold with changing neural network architectures. We further demonstrate how the complexity of linear regions is different on the low dimensional manifold of images as compared to the Euclidean space, using the MetFaces dataset.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Builds on 56 of its 87 references that are in the map. First posted on arXiv 29 Dec 2022.

Builds on

56 of 87 references.

References · 56

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,282
  3. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  4. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 5,986
  5. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,223
  6. 2013Intriguing properties of neural networkscited by 5,114
  7. 2019Analyzing and Improving the Image Quality of StyleGANcited by 3,580
  8. 2019A Comprehensive Survey on Graph Neural Networkscited by 2,906
  9. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  10. 2018Dynamic Graph CNN for Learning on Point Cloudscited by 2,220
  11. 2016Understanding deep learning requires rethinking generalizationcited by 1,657
  12. 2018On the Spectral Bias of Neural Networkscited by 1,321
38 more
  1. 2020Training Generative Adversarial Networks with Limited Datacited by 1,184
  2. 2021Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gaugescited by 1,131
  3. 2016Geometric deep learning: going beyond Euclidean datacited by 1,042
  4. 2017A Closer Look at Memorization in Deep Networkscited by 936
  5. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 784
  6. 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 748
  7. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 732
  8. 2016Geometric deep learning on graphs and manifolds using mixture model CNNscited by 468
  9. 2019Implicit Regularization in Deep Matrix Factorizationcited by 402
  10. 2018Hyperbolic Neural Networkscited by 393
  11. 2014On the Number of Linear Regions of Deep Neural Networkscited by 378
  12. 2017A PAC-Bayesian Approach to Spectrally-Normalized Margin Bounds for Neural Networkscited by 309
  13. 2018Stronger generalization bounds for deep nets via a compression approachcited by 306
  14. 2016On the Expressive Power of Deep Neural Networkscited by 302
  15. 2017Empirical Analysis of the Hessian of Over-Parametrized Neural Networkscited by 283
  16. 2016Exponential expressivity in deep neural networks through transient chaoscited by 260
  17. 2015Deep Convolutional Networks on Graph-Structured Datacited by 246
  18. 2015A Differential Equation for Modeling Nesterov's Accelerated Gradient Method: Theory and Insightscited by 203
  19. 2018Sensitivity and Generalization in Neural Networks: an Empirical Studycited by 187
  20. 2014Qualitatively characterizing neural network optimization problemscited by 186
  21. 2018A Convergence Analysis of Gradient Descent for Deep Linear Neural Networkscited by 180
  22. 2018Algorithmic Regularization in Learning Deep Homogeneous Models: Layers are Automatically Balancedcited by 172
  23. 2019Complexity of Linear Regions in Deep Networkscited by 166
  24. 2017Bounding and Counting Linear Regions of Deep Neural Networkscited by 161
  25. 2019Deep ReLU Networks Have Surprisingly Few Activation Patternscited by 160
  26. 2017Generalization in Deep Learningcited by 146
  27. 2017A Bayesian Perspective on Generalization and Stochastic Gradient Descentcited by 134
  28. 2020Flows for simultaneous manifold learning and density estimationcited by 117
  29. 2019Finite Depth and Width Corrections to the Neural Tangent Kernelcited by 114
  30. 2017Universal Function Approximation by Deep Neural Nets with Bounded Width and ReLU Activationscited by 104
  31. 2017Fisher-Rao Metric, Geometry, and Complexity of Neural Networkscited by 99
  32. 2020Neural Networks Fail to Learn Periodic Functions and How to Fix Itcited by 97
  33. 2020Neural Mechanics: Symmetry and Broken Conservation Laws in Deep Learning Dynamicscited by 93
  34. 2015Representation Benefits of Deep Feedforward Networkscited by 84
  35. 2019Deep ReLU network approximation of functions on a manifoldcited by 73
  36. 2016Deep vs. shallow networks : An approximation theory perspectivecited by 63
  37. 2015Provable approximation properties for deep neural networkscited by 60
  38. 2018Products of Many Large Random Matrices and Gradients in Deep Neural Networkscited by 49

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 2; 2025: 2; 2026: 5.

12 papers.

  1. 2023When Deep Learning Meets Polyhedral Theory: A Surveycited by 39
  2. 2025Geometry of Neural Reinforcement Learning in Continuous State and Action Spacescited by 3
  3. 2024Defining Neural Network Architecture through Polytope Structures of Datasetcited by 2
  4. 2025Generalization Below the Edge of Stability: The Role of Data Geometrycited by 2
  5. 2023Data Topology-Dependent Upper Bounds of Neural Network Widthscited by 1
  6. 2023On original and latent space connectivity in deep neural networkscited by 1
  7. 2026Training-Time Batch Normalization Reshapes Local Partition Geometry in Piecewise-Affine Networkscited by 1
  8. 2026AffineLens: Capturing the Continuous Piecewise Affine Functions of Neural Networkscited by 1
  9. 2026Region Seeding via Pre-Activation Regularization: A Geometric View of Piecewise Affine Neural Networkscited by 1
  10. 2026The Geometry of Projection Heads: Conditioning, Invariance, and Collapsecited by 1
  11. 2024Data-induced multiscale losses and efficient multirate gradient descent schemescited by 0
  12. 2026Expressivity Saturation: Reduced Affine Region Usage Under Increasing Task Complexitycited by 0