Hamiltonian Mechanics of Feature Learning: Bottleneck Structure in Leaky ResNets
Not on the map. Why?
Abstract (arXiv)
We study Leaky ResNets, which interpolate between ResNets and Fully-Connected nets depending on an 'effective depth' hyper-parameter tildeL. In the infinite depth limit, we study 'representation geodesics' Ap: continuous paths in representation space (similar to NeuralODEs) from input p=0 to output p=1 that minimize the parameter norm of the network. We give a Lagrangian and Hamiltonian reformulation, which highlight the importance of two terms: a kinetic energy which favors small layer derivatives partialpAp and a potential energy that favors low-dimensional representations, as measured by the 'Cost of Identity'. The balance between these two forces offers an intuitive understanding of feature learning in ResNets. We leverage this intuition to explain the emergence of a bottleneck structure, as observed in previous work: for large tildeL the potential energy dominates and leads to a separation of timescales, where the representation jumps rapidly from the high dimensional inputs to a low-dimensional representation, move slowly inside the space of low-dimensional representations, before jumping back to the potentially high-dimensional outputs. Inspired by this phenomenon, we train with an adaptive layer step-size to adapt to the separation of timescales.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 24 of its 33 references that are in the map. First posted on arXiv 27 May 2024.
Builds on
24 of 33 references.
References · 24
- 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,375
- 2018Neural Ordinary Differential Equationscited by 3,779
- 2013Visualizing and Understanding Convolutional Networkscited by 2,290
- 2019Similarity of Neural Network Representations Revisitedcited by 1,755
- 2015Deep Learning and the Information Bottleneck Principlecited by 802
- 2020Prevalence of Neural Collapse during the terminal phase of deep learning trainingcited by 719
- 2014Breaking the Curse of Dimensionality with Convex Neural Networkscited by 331
- 2020Implicit Bias of Gradient Descent for Wide Two-layer Neural Networks Trained with the Logistic Losscited by 268
- 2018Characterizing Implicit Bias in Terms of Optimization Geometrycited by 266
- 2018Implicit Bias of Gradient Descent on Linear Convolutional Networkscited by 251
- 2020Towards Resolving the Implicit Bias of Gradient Descent for Matrix Factorization: Greedy Low-Rank Learningcited by 150
- 2022The merged-staircase property: a necessary and nearly sufficient condition for SGD learning of sparse functions on two-layer neural networkscited by 142
12 more
- 2021The staircase property: How hierarchical structure can guide deep learningcited by 62
- 2018Towards Understanding Linear Word Analogiescited by 57
- 2023Deep Neural Collapse Is Provably Optimal for the Deep Unconstrained Features Modelcited by 43
- 2022Implicit Bias of Large Depth Networks: a Notion of Rank for Nonlinear Functionscited by 38
- 2023Implicit bias of SGD in L2-regularized linear DNNs: One-way jumps from high to low rankcited by 29
- 2023A Rainbow in Deep Network Black Boxescited by 18
- 2022Feature Learning in L2-regularized DNNs: Attraction/Repulsion and Sparsitycited by 17
- 2023Bottleneck Structure in Learned Features: Low-Dimension vs Regularity Tradeoffcited by 17
- 2023Mechanism of feature learning in convolutional neural networkscited by 17
- 2024Residual Alignment: Uncovering the Mechanisms of Residual Networkscited by 13
- 2022Turnpike in optimal control of PDEs, ResNets, and beyondcited by 11
- 2024Which Frequencies do CNNs Need? Emergent Bottleneck Structure in Feature Learningcited by 9
Built on by
Citing papers per year: 2025: 1.
1 paper.