Geometric Inductive Biases of Deep Networks: The Role of Data and Architecture
Not on the map. Why?
Abstract (arXiv)
In this paper, we propose the textitgeometric invariance hypothesis (GIH), which argues that the input space curvature of a neural network remains invariant under transformation in certain architecture-dependent directions during training. We investigate a simple, non-linear binary classification problem residing on a plane in a high dimensional space and observe thatunicodex2014unlike MLPsunicodex2014ResNets fail to generalize depending on the orientation of the plane. Motivated by this example, we define a neural network's average geometry and average geometry evolution as compact textitarchitecture-dependent summaries of the model's input-output geometry and its evolution during training. By investigating the average geometry evolution at initialization, we discover that the geometry of a neural network evolves according to the data covariance projected onto its average geometry. This means that the geometry only changes in a subset of the input space when the average geometry is low-rank, such as in ResNets. This causes an architecture-dependent invariance property in the input space curvature, which we dub GIH. Finally, we present extensive experimental results to observe the consequences of GIH and how it relates to generalization in neural networks.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 39 of its 49 references that are in the map. First posted on arXiv 15 Oct 2024.
Builds on
39 of 49 references.
References · 39
- 2017Attention Is All You Needcited by 60,361
- 2015Deep Residual Learning for Image Recognitioncited by 52,096
- 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
- 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
- 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
- 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,415
- 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,718
- 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,548
- 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
- 2016Understanding deep learning requires rethinking generalizationcited by 1,674
- 2018Relational inductive biases, deep learning, and graph networkscited by 1,360
- 2015DeepFool: a simple and accurate method to fool deep neural networkscited by 1,318
27 more
- 2017A Closer Look at Memorization in Deep Networkscited by 942
- 2020Efficient Transformers: A Surveycited by 790
- 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 738
- 2017The Implicit Bias of Gradient Descent on Separable Datacited by 655
- 2017Exploring Generalization in Deep Learningcited by 600
- 2014New insights and perspectives on the natural gradient methodcited by 443
- 2019Implicit Regularization in Deep Matrix Factorizationcited by 405
- 2020The Pitfalls of Simplicity Bias in Neural Networkscited by 351
- 2019Uniform convergence may be unable to explain generalization in deep learningcited by 224
- 2019SGD on Neural Networks Learns Functions of Increasing Complexitycited by 204
- 2021On the Origin of Implicit Regularization in Stochastic Gradient Descentcited by 184
- 2018Deep learning generalizes because the parameter-function map is biased towards simple functionscited by 183
- 2019On the Inductive Bias of Neural Tangent Kernelscited by 177
- 2020Implicit Gradient Regularizationcited by 146
- 2023Neural Architecture Search: Insights from 1000 Paperscited by 139
- 2019The Convergence Rate of Neural Networks for Learned Functions of Different Frequenciescited by 135
- 2021Vision Transformer for Small-Size Datasetscited by 122
- 2021Neural Networks as Kernel Learners: The Silent Alignment Effectcited by 106
- 2022Scaling Laws vs Model Architectures: How does Inductive Bias Influence Scaling?cited by 103
- 2021What can linearized neural networks actually say about generalization?cited by 49
- 2024Neural Redshift: Random Networks are not Random Functionscited by 43
- 2023Beyond NTK with Vanilla Gradient Descent: A Mean-Field Analysis of Neural Networks with Polynomial Width, Samples, and Timecited by 20
- 2024Mixed Dynamics In Linear Networks: Unifying the Lazy and Active Regimescited by 18
- 2020Neural Anisotropy Directionscited by 17
- 2024Which Frequencies do CNNs Need? Emergent Bottleneck Structure in Feature Learningcited by 9
- 2021Uniform Convergence, Adversarial Spheres and a Simple Remedycited by 8
- 2023No Wrong Turns: The Simple Geometry Of Neural Networks Optimization Pathscited by 7
Built on by
Citing papers per year: 2025: 2.
2 papers.