mlsanityData to 7 Oct 2026

Transition to Linearity of General Neural Networks with Directed Acyclic Graph Architecture

arXiv 2205.11786 ↗First posted 24 May 2022cs.LG math.OC stat.MLcited by 4

Not on the map. Why?

Abstract (arXiv)

In this paper we show that feedforward neural networks corresponding to arbitrary directed acyclic graphs undergo transition to linearity as their "width" approaches infinity. The width of these general networks is characterized by the minimum in-degree of their neurons, except for the input and first layers. Our results identify the mathematical structure underlying transition to linearity and generalize a number of recent works aimed at characterizing transition to linearity or constancy of the Neural Tangent Kernel for standard architectures.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 23 of its 46 references that are in the map. First posted on arXiv 24 May 2022.

Builds on

23 of 46 references.

References · 23

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2016Densely Connected Convolutional Networkscited by 6,691
  3. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  4. 2014Going Deeper with Convolutionscited by 4,231
  5. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  6. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 784
  7. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 732
  8. 2018On Lazy Training in Differentiable Programmingcited by 671
  9. 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 655
  10. 2018Gradient Descent Finds Global Minima of Deep Neural Networkscited by 572
  11. 2019On Exact Computation with an Infinitely Wide Neural Netcited by 572
  12. 2015An Introduction to Matrix Concentration Inequalitiescited by 391
11 more
  1. 2020Loss landscapes and optimization in over-parameterized non-linear systems and neural networkscited by 231
  2. 2020On the linearity of large non-linear models: when and why the tangent kernel is constantcited by 129
  3. 2019An Improved Analysis of Training Over-parameterized Deep Neural Networkscited by 120
  4. 2020Infinite attention: NNGP and NTK for deep attention networkscited by 117
  5. 2019Finite Depth and Width Corrections to the Neural Tangent Kernelcited by 114
  6. 2020Tight Bounds on the Smallest Eigenvalue of the Neural Tangent Kernel for Deep ReLU Networkscited by 87
  7. 2020The Interpolation Phase Transition in Neural Networks: Memorization and Generalization under Lazy Trainingcited by 84
  8. 2020The Recurrent Neural Tangent Kernelcited by 67
  9. 2020Graph Structure of Neural Networkscited by 67
  10. 2019Discovering Neural Wiringscited by 56
  11. 2022Transition to Linearity of Wide Neural Networks is an Emerging Property of Assembling Weak Modelscited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 0; 2025: 0; 2026: 1.

4 papers.

  1. 2023Catapults in SGD: spikes in the training loss and their impact on generalization through feature learningcited by 29
  2. 2023Toward Large Kernel Modelscited by 12
  3. 2023On Emergence of Clean-Priority Learning in Early Stopped Neural Networkscited by 3
  4. 2026Label-NTK Alignments and A Tighter Convergence Bound in the NTK Regimecited by 0