mlsanityData to 14 Oct 2026

Transition to Linearity of Wide Neural Networks is an Emerging Property of Assembling Weak Models

arXiv 2203.05104 ↗First posted 10 Mar 2022cs.LGcited by 6

Not on the map. Why?

Abstract (arXiv)

Wide neural networks with linear output layer have been shown to be near-linear, and to have near-constant neural tangent kernel (NTK), in a region containing the optimization path of gradient descent. These findings seem counter-intuitive since in general neural networks are highly complex models. Why does a linear structure emerge when the networks become wide? In this work, we provide a new perspective on this "transition to linearity" by considering a neural network as an assembly model recursively built from a set of sub-models corresponding to individual neurons. In this view, we show that the linearity of wide neural networks is, in fact, an emerging property of assembling a large number of diverse "weak" sub-models, none of which dominate the assembly.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 10 of its 18 references that are in the map. First posted on arXiv 10 Mar 2022.

Builds on

10 of 18 references.

References · 10

  1. 2016Densely Connected Convolutional Networkscited by 6,720
  2. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  3. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 787
  4. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 738
  5. 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 659
  6. 2018Gradient Descent Finds Global Minima of Deep Neural Networkscited by 574
  7. 2020Loss landscapes and optimization in over-parameterized non-linear systems and neural networkscited by 233
  8. 2019Towards moderate overparameterization: global convergence guarantees for training shallow neural networkscited by 176
  9. 2018Stochastic Gradient Descent Optimizes Over-parameterized Deep ReLU Networkscited by 141
  10. 2020On the linearity of large non-linear models: when and why the tangent kernel is constantcited by 129

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 1; 2023: 2; 2024: 2; 2025: 0; 2026: 1.

6 papers.

  1. 2023Catapults in SGD: spikes in the training loss and their impact on generalization through feature learningcited by 31
  2. 2022Transition to Linearity of General Neural Networks with Directed Acyclic Graph Architecturecited by 4
  3. 2023On Emergence of Clean-Priority Learning in Early Stopped Neural Networkscited by 3
  4. 2024On the Role of Initialization on the Implicit Bias in Deep Linear Networkscited by 2
  5. 2024Weak Correlations as the Underlying Principle for Linearization of Gradient-Based Learning Systemscited by 0
  6. 2026Label-NTK Alignments and A Tighter Convergence Bound in the NTK Regimecited by 0