# Random weights of DNNs and emergence of fixed points

arXiv: https://arxiv.org/abs/2501.04182 · HTML: https://mlsanity.com/paper/2501.04182/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 38 references that are in the map.
- First posted on arXiv 7 Jan 2025.

## Builds on (most cited first, 16 of 16)

- [An Introduction to Variational Autoencoders](https://mlsanity.com/paper/1906.02691/) (2019)
- [Gaussian Process Behaviour in Wide Deep Neural Networks](https://mlsanity.com/paper/1804.11271/) (2018)
- [Implicit Self-Regularization in Deep Neural Networks: Evidence from Random Matrix Theory and Implications for Learning](https://mlsanity.com/paper/1810.01075/) (2018)
- [Robust Learning with Jacobian Regularization](https://mlsanity.com/paper/1908.02729/) (2019)
- [Deep Generalized Unfolding Networks for Image Restoration](https://mlsanity.com/paper/2204.13348/) (2022)
- [Large Associative Memory Problem in Neurobiology and Machine Learning](https://mlsanity.com/paper/2008.06996/) (2020)
- [The Emergence of Spectral Universality in Deep Networks](https://mlsanity.com/paper/1802.09979/) (2018)
- [Tensor Programs III: Neural Matrix Laws](https://mlsanity.com/paper/2009.10685/) (2020)
- [Deep Randomized Neural Networks](https://mlsanity.com/paper/2002.12287/) (2020)
- [On Random Matrices Arising in Deep Neural Networks: General I.I.D. Case](https://mlsanity.com/paper/2011.11439/) (2020)
- [Fixed points of nonnegative neural networks](https://mlsanity.com/paper/2106.16239/) (2021)
- [Enhancing Accuracy in Deep Learning Using Random Matrix Theory](https://mlsanity.com/paper/2310.03165/) (2023)
- [Spectrum concentration in deep residual learning: a free probability approach](https://mlsanity.com/paper/1807.11694/) (2018)
- [Understanding Gradient Descent through the Training Jacobian](https://mlsanity.com/paper/2412.07003/) (2024)
- [Fixed points of competitive threshold-linear networks](https://mlsanity.com/paper/1804.00794/) (2018)
- [Deep Learning Weight Pruning with RMT-SVD: Increasing Accuracy and Reducing Overfitting](https://mlsanity.com/paper/2303.08986/) (2023)

## Built on by (most cited first, 1 of 1)

- [Randomly initialized autoencoders: fixed points and edge-of-chaos](https://mlsanity.com/paper/2608.14638/) (2026)

## Abstract (arXiv)

This paper is concerned with a special class of deep neural networks (DNNs) where the input and the output vectors have the same dimension. Such DNNs are widely used in applications, e.g., autoencoders. The training of such networks can be characterized by their fixed points (FPs). We are concerned with the dependence of the FPs number and their stability on the distribution of randomly initialized DNNs' weight matrices. Specifically, we consider the i.i.d. random weights with heavy and light-tail distributions. Our objectives are twofold. First, the dependence of FPs number and stability of FPs on the type of the distribution tail. Second, the dependence of the number of FPs on the DNNs' architecture. We perform extensive simulations and show that for light tails (e.g., Gaussian), which are typically used for initialization, a single stable FP exists for broad types of architectures. In contrast, for heavy tail distributions (e.g., Cauchy), which typically appear in trained DNNs, a number of FPs emerge. We further observe that these FPs are stable attractors and their basins of attraction partition the domain of input vectors. Finally, we observe an intriguing non-monotone dependence of the number of fixed points Q(L) on the DNNs' depth L. The above results were first obtained for untrained DNNs with two types of distributions at initialization and then verified by considering DNNs in which the heavy tail distributions arise in training.