mlsanityData to 7 Oct 2026

Deep Layer-wise Networks Have Closed-Form Weights

arXiv 2202.01210 ↗First posted 1 Feb 2022stat.ML cs.LG math.STcited by 2

Not on the map. Why?

Abstract (arXiv)

There is currently a debate within the neuroscience community over the likelihood of the brain performing backpropagation (BP). To better mimic the brain, training a network textitone layer at a time with only a "single forward pass" has been proposed as an alternative to bypass BP; we refer to these networks as "layer-wise" networks. We continue the work on layer-wise networks by answering two outstanding questions. First, textitdo they have a closed-form solution? Second, textithow do we know when to stop adding more layers? This work proves that the Kernel Mean Embedding is the closed-form weight that achieves the network global optimum while driving these networks to converge towards a highly desirable kernel for classification; we call it the textitNeural Indicator Kernel.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 34 of its 64 references that are in the map. First posted on arXiv 1 Feb 2022.

Builds on

34 of 64 references.

References · 34

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  3. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
  4. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  5. 2016Understanding deep learning requires rethinking generalizationcited by 1,657
  6. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,242
  7. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 732
  8. 2017Deep Neural Networks as Gaussian Processescited by 617
  9. 2019On Exact Computation with an Infinitely Wide Neural Netcited by 572
  10. 2016Kernel Mean Embedding of Distributions: A Review and Beyondcited by 360
  11. 2018Learning and Generalization in Overparameterized Neural Networks, Going Beyond Two Layerscited by 340
  12. 2018Gaussian Process Behaviour in Wide Deep Neural Networkscited by 314
22 more
  1. 2019Intrinsic dimension of data representations in deep neural networkscited by 276
  2. 2019Generalization Bounds of Stochastic Gradient Descent for Wide and Deep Neural Networkscited by 221
  3. 2018Universality of Deep Convolutional Neural Networkscited by 176
  4. 2019Scaling Limits of Wide Neural Networks with Weight Sharing: Gaussian Process Behavior, Gradient Independence, and Neural Tangent Kernel Derivationcited by 146
  5. 2019Training Neural Networks with Local Error Signalscited by 135
  6. 2018Greedy Layerwise Learning Can Scale to ImageNetcited by 113
  7. 2019On the Impact of the Activation Function on Deep Neural Networks Trainingcited by 101
  8. 2018Dendritic cortical microcircuits approximate the backpropagation algorithmcited by 96
  9. 2019Harnessing the Power of Infinitely Wide Deep Nets on Small-data Taskscited by 87
  10. 2019Putting An End to End-to-End: Gradient-Isolated Learning of Representationscited by 85
  11. 2017SGD Learns Over-parameterized Networks that Provably Generalize on Linearly Separable Datacited by 79
  12. 2019The HSIC Bottleneck: Deep Learning without Back-Propagationcited by 77
  13. 2016Towards deep learning with segregated dendritescited by 71
  14. 2015Towards Biologically Plausible Deep Learningcited by 66
  15. 2015How Important is Weight Symmetry in Backpropagation?cited by 57
  16. 2014Convolutional Kernel Networkscited by 47
  17. 2014Avoiding pathologies in very deep networkscited by 33
  18. 2020Kernelized information bottleneck leads to biologically plausible 3-factor Hebbian learning in deep networkscited by 25
  19. 2017Layer-wise training of deep networks using kernel similaritycited by 15
  20. 2020Learning to Learn with Feedback and Local Plasticitycited by 11
  21. 2018On Kernel Method-Based Connectionist Models and Supervised Deep Learning Without Backpropagationcited by 4
  22. 2019Iterative Spectral Method for Alternative Clusteringcited by 2

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 0; 2025: 1.

2 papers.

  1. 2023Convolutional Deep Kernel Machinescited by 3
  2. 2025Information-Theoretic Greedy Layer-wise Training for Traffic Sign Recognitioncited by 0