mlsanityData to 7 Oct 2026

Eigenvalues of Autoencoders in Training and at Initialization

arXiv 2201.11813 ↗First posted 27 Jan 2022cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

In this paper, we investigate the evolution of autoencoders near their initialization. In particular, we study the distribution of the eigenvalues of the Jacobian matrices of autoencoders early in the training process, training on the MNIST data set. We find that autoencoders that have not been trained have eigenvalue distributions that are qualitatively different from those which have been trained for a long time (>100 epochs). Additionally, we find that even at early epochs, these eigenvalue distributions rapidly become qualitatively similar to those of the fully trained autoencoders. We also compare the eigenvalues at initialization to pertinent theoretical work on the eigenvalues of random matrices and the products of such matrices.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 12 of its 31 references that are in the map. First posted on arXiv 27 Jan 2022.

Builds on

12 of 31 references.

References · 12

  1. 2019An Investigation into Neural Net Optimization via Hessian Eigenvalue Densitycited by 289
  2. 2017Empirical Analysis of the Hessian of Over-Parametrized Neural Networkscited by 283
  3. 2019Intrinsic dimension of data representations in deep neural networkscited by 276
  4. 2018Estimating the intrinsic dimension of datasets by a minimal neighborhood informationcited by 246
  5. 2016Direct Feedback Alignment Provides Learning in Deep Neural Networkscited by 240
  6. 2020How Neural Networks Extrapolate: From Feedforward to Graph Neural Networkscited by 218
  7. 2018Gradient Descent Happens in a Tiny Subspacecited by 208
  8. 2016Eigenvalues of the Hessian in Deep Learning: Singularity and Beyondcited by 156
  9. 2020The Early Phase of Neural Network Trainingcited by 114
  10. 2020Random Vector Functional Link Networks for Function Approximation on Manifoldscited by 19
  11. 2021Random vector functional link neural network based ensemble deep learning for short-term load forecastingcited by 5
  12. 2021Geometry and Generalization: Eigenvalues as predictors of where a network will fail to generalizecited by 2

Built on by

Not cited yet.