mlsanityData to 14 Oct 2026

The kernel-balanced equation for deep neural networks

arXiv 2309.07367 ↗First posted 14 Sep 2023cond-mat.dis-nn cs.AI cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Deep neural networks have shown many fruitful applications in this decade. A network can get the generalized function through training with a finite dataset. The degree of generalization is a realization of the proximity scale in the data space. Specifically, the scale is not clear if the dataset is complicated. Here we consider a network for the distribution estimation of the dataset. We show the estimation is unstable and the instability depends on the data density and training duration. We derive the kernel-balanced equation, which gives a short phenomenological description of the solution. The equation tells us the reason for the instability and the mechanism of the scale. The network outputs a local average of the dataset as a prediction and the scale of averaging is determined along the equation. The scale gradually decreases along training and finally results in instability in our case.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 11 of its 35 references that are in the map. First posted on arXiv 14 Sep 2023.

Builds on

11 of 35 references.

References · 11

  1. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  2. 2019Aleatoric and Epistemic Uncertainty in Machine Learning: An Introduction to Concepts and Methodscited by 913
  3. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 738
  4. 2021A Survey of Uncertainty in Deep Neural Networkscited by 680
  5. 2020Bayesian Deep Learning and a Probabilistic Perspective of Generalizationcited by 480
  6. 2019Self-Driving Cars: A Surveycited by 148
  7. 2019Reliable training and estimation of variance networkscited by 63
  8. 2020Analyzing Finite Neural Networks: Can We Trust Neural Tangent Kernel Theory?cited by 33
  9. 2022Neural Tangent Kernel Beyond the Infinite-Width Limit: Effects of Depth and Initializationcited by 30
  10. 2022The training response law explains how deep neural networks learncited by 1
  11. 2023Measuring and Mitigating Local Instability in Deep Neural Networkscited by 1

Built on by

Not cited yet.