mlsanityData to 7 Oct 2026

Feature Learning and Signal Propagation in Deep Neural Networks

arXiv 2110.11749 ↗First posted 22 Oct 2021stat.ML cs.LGcited by 18

Not on the map. Why?

Abstract (arXiv)

Recent work by Baratin et al. (2021) sheds light on an intriguing pattern that occurs during the training of deep neural networks: some layers align much more with data compared to other layers (where the alignment is defined as the euclidean product of the tangent features matrix and the data labels matrix). The curve of the alignment as a function of layer index (generally) exhibits an ascent-descent pattern where the maximum is reached for some hidden layer. In this work, we provide the first explanation for this phenomenon. We introduce the Equilibrium Hypothesis which connects this alignment pattern to signal propagation in deep neural networks. Our experiments demonstrate an excellent match with the theoretical predictions.

Key facts. Cited by 18 papers in the map (data through 7 Oct 2026). Builds on 28 of its 42 references that are in the map. First posted on arXiv 22 Oct 2021.

Builds on

28 of 42 references.

References · 28

  1. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
  2. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  3. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,479
  4. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
  5. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 732
  6. 2017Deep Neural Networks as Gaussian Processescited by 617
  7. 2018Gaussian Process Behaviour in Wide Deep Neural Networkscited by 314
  8. 2017A PAC-Bayesian Approach to Spectrally-Normalized Margin Bounds for Neural Networkscited by 309
  9. 2019An Investigation into Neural Net Optimization via Hessian Eigenvalue Densitycited by 289
  10. 2016Exponential expressivity in deep neural networks through transient chaoscited by 260
  11. 2016Deep Information Propagationcited by 194
  12. 2018Deep learning generalizes because the parameter-function map is biased towards simple functionscited by 180
16 more
  1. 2018Dynamical Isometry and a Mean Field Theory of CNNs: How to Train 10,000-Layer Vanilla Convolutional Neural Networkscited by 176
  2. 2020Finite Versus Infinite Neural Networks: an Empirical Studycited by 156
  3. 2019Tensor Programs I: Wide Feedforward or Recurrent Neural Networks of Any Architecture are Gaussian Processescited by 148
  4. 2019Scaling Limits of Wide Neural Networks with Weight Sharing: Gaussian Process Behavior, Gradient Independence, and Neural Tangent Kernel Derivationcited by 146
  5. 2019On the Impact of the Activation Function on Deep Neural Networks Trainingcited by 101
  6. 2018Universal Statistics of Fisher Information in Deep Neural Networks: Mean Field Approachcited by 89
  7. 2017Towards Understanding Generalization of Deep Learning: Perspective of Loss Landscapescited by 84
  8. 2017Mean Field Residual Networks: On the Edge of Chaoscited by 83
  9. 2019Generalization Guarantees for Neural Networks via Harnessing the Low-rank Structure of the Jacobiancited by 67
  10. 2021Stochastic Training is Not Necessary for Generalizationcited by 63
  11. 2018The jamming transition as a paradigm to understand the loss landscape of deep neural networkscited by 61
  12. 2020Rethinking Parameter Counting in Deep Models: Effective Dimensionality Revisitedcited by 50
  13. 2020Is SGD a Bayesian sampler? Well, almostcited by 50
  14. 2020Tensor Programs III: Neural Matrix Lawscited by 47
  15. 2019The asymptotic spectrum of the Hessian of DNN throughout trainingcited by 19
  16. 2017Layer-wise training of deep networks using kernel similaritycited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 3; 2023: 7; 2024: 4; 2025: 2; 2026: 2.

18 papers.

Sort:

  1. 2022Self-Consistent Dynamical Field Theory of Kernel Evolution in Wide Neural Networkscited by 104
  2. 2023Dynamics of Finite Width Kernel and Prediction Fluctuations in Mean Field Neural Networkscited by 42
  3. 2023The Tunnel Effect: Building Data Representations in Deep Neural Networkscited by 34
  4. 2022On the infinite-depth limit of finite-width neural networkscited by 22
  5. 2023Width and Depth Limits Commute in Residual Networkscited by 21
  6. 2023Leave-one-out Distinguishability in Machine Learningcited by 20
  7. 2022The Influence of Learning Rule on Representation Dynamics in Wide Neural Networkscited by 19
  8. 2023Neural (Tangent Kernel) Collapsecited by 18
  9. 2025PLoP: Precise LoRA Placement for Efficient Finetuning of Large Modelscited by 13
  10. 2023The Feature Speed Formula: a flexible approach to scale hyper-parameters of deep neural networkscited by 11
  11. 2024Understanding and Minimising Outlier Features in Neural Network Trainingcited by 10
  12. 2024Theoretical characterisation of the Gauss-Newton conditioning in Neural Networkscited by 5
6 more
  1. 2024Spring-block theory of feature learning in deep neural networkscited by 2
  2. 2024Decoupling Dynamical Richness from Representation Learning: Towards Practical Measurementcited by 2
  3. 2025GradPCA: Leveraging NTK Alignment for Reliable Out-of-Distribution Detectioncited by 2
  4. 2023Commutative Width and Depth Scaling in Deep Neural Networkscited by 1
  5. 2026Noise Stability of Transformer Modelscited by 0
  6. 2026Q-TriM: Question-Guided Tri-Modal Attention for Audio-Visual Question Answeringcited by 0