# The kernel-balanced equation for deep neural networks

arXiv: https://arxiv.org/abs/2309.07367 · HTML: https://mlsanity.com/paper/2309.07367/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 11 of its 35 references that are in the map.
- First posted on arXiv 14 Sep 2023.

## Builds on (most cited first, 11 of 11)

- [Neural Tangent Kernel: Convergence and Generalization in Neural Networks](https://mlsanity.com/paper/1806.07572/) (2018)
- [Aleatoric and Epistemic Uncertainty in Machine Learning: An Introduction to Concepts and Methods](https://mlsanity.com/paper/1910.09457/) (2019)
- [Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descent](https://mlsanity.com/paper/1902.06720/) (2019)
- [A Survey of Uncertainty in Deep Neural Networks](https://mlsanity.com/paper/2107.03342/) (2021)
- [Bayesian Deep Learning and a Probabilistic Perspective of Generalization](https://mlsanity.com/paper/2002.08791/) (2020)
- [Self-Driving Cars: A Survey](https://mlsanity.com/paper/1901.04407/) (2019)
- [Reliable training and estimation of variance networks](https://mlsanity.com/paper/1906.03260/) (2019)
- [Analyzing Finite Neural Networks: Can We Trust Neural Tangent Kernel Theory?](https://mlsanity.com/paper/2012.04477/) (2020)
- [Neural Tangent Kernel Beyond the Infinite-Width Limit: Effects of Depth and Initialization](https://mlsanity.com/paper/2202.00553/) (2022)
- [The training response law explains how deep neural networks learn](https://mlsanity.com/paper/2204.07291/) (2022)
- [Measuring and Mitigating Local Instability in Deep Neural Networks](https://mlsanity.com/paper/2305.10625/) (2023)

## Abstract (arXiv)

Deep neural networks have shown many fruitful applications in this decade. A network can get the generalized function through training with a finite dataset. The degree of generalization is a realization of the proximity scale in the data space. Specifically, the scale is not clear if the dataset is complicated. Here we consider a network for the distribution estimation of the dataset. We show the estimation is unstable and the instability depends on the data density and training duration. We derive the kernel-balanced equation, which gives a short phenomenological description of the solution. The equation tells us the reason for the instability and the mechanism of the scale. The network outputs a local average of the dataset as a prediction and the scale of averaging is determined along the equation. The scale gradually decreases along training and finally results in instability in our case.