# Regularized linear autoencoders recover the principal components, eventually

arXiv: https://arxiv.org/abs/2007.06731 · HTML: https://mlsanity.com/paper/2007.06731/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Jul 2020.

## Built on by (most cited first, 26 of 26)

- [The dynamics of representation learning in shallow, non-linear autoencoders](https://mlsanity.com/paper/2201.02115/) (2022)
- [Bayesian autoencoders for data-driven discovery of coordinates, governing equations and fundamental constants](https://mlsanity.com/paper/2211.10575/) (2022)
- [Sparse identification of nonlinear dynamics and Koopman operators with Shallow Recurrent Decoder Networks](https://mlsanity.com/paper/2501.13329/) (2025)
- [High-dimensional Asymptotics of Denoising Autoencoders](https://mlsanity.com/paper/2305.11041/) (2023)
- [Analysis of feature learning in weight-tied autoencoders via the mean field lens](https://mlsanity.com/paper/2102.08373/) (2021)
- [When does Self-Prediction help? Understanding Auxiliary Tasks in Reinforcement Learning](https://mlsanity.com/paper/2406.17718/) (2024)
- [Identifiability of Deep Polynomial Neural Networks](https://mlsanity.com/paper/2506.17093/) (2025)
- [Anytime Sampling for Autoregressive Models via Ordered Autoencoding](https://mlsanity.com/paper/2102.11495/) (2021)
- [Neural Characteristic Activation Analysis and Geometric Parameterization for ReLU Networks](https://mlsanity.com/paper/2305.15912/) (2023)
- [Compression of Structured Data with Autoencoders: Provable Benefit of Nonlinearities and Depth](https://mlsanity.com/paper/2402.05013/) (2024)
- [Good Things Come in Pairs: Paired Autoencoders for Inverse Problems](https://mlsanity.com/paper/2505.06549/) (2025)
- [Fundamental Limits of Two-layer Autoencoders, and Achieving Them with Gradient Methods](https://mlsanity.com/paper/2212.13468/) (2022)
- [The Effect of SGD Batch Size on Autoencoder Learning: Sparsity, Sharpness, and Feature Learning](https://mlsanity.com/paper/2308.03215/) (2023)
- [Escaping Plato's Cave: JAM for Aligning Independently Trained Vision and Language Models](https://mlsanity.com/paper/2507.01201/) (2025)
- [PCAE: Learning Ordered Representations in Latent Space for Intrinsic Dimension Estimation via Principal Component Autoencoder](https://mlsanity.com/paper/2601.19179/) (2026)
- [Principal Bit Analysis: Autoencoding with Schur-Concave Loss](https://mlsanity.com/paper/2106.02796/) (2021)
- [Improved Representation Learning Through Tensorized Autoencoders](https://mlsanity.com/paper/2212.01046/) (2022)
- [VAE-DNN: Energy-Efficient Trainable-by-Parts Surrogate Model For Parametric Partial Differential Equations](https://mlsanity.com/paper/2508.03839/) (2025)
- [ODBAE: a high-performance model identifying complex phenotypes in high-dimensional biological datasets](https://mlsanity.com/paper/2211.03054/) (2022)
- [Disentanglement as Identifiable Pushforward Factorisation](https://mlsanity.com/paper/2410.22559/) (2024)
- [Optimal Linear Baseline Models for Scientific Machine Learning](https://mlsanity.com/paper/2508.05831/) (2025)
- [Theoretical Foundations of Representation Learning using Unlabeled Data: Statistics and Optimization](https://mlsanity.com/paper/2509.18997/) (2025)
- [Latent Dynamics Graph Convolutional Networks for model order reduction of parameterized time-dependent PDEs](https://mlsanity.com/paper/2601.11259/) (2026)
- [NeuraLSP: A Neural Spectral Preconditioner for Accelerating PDE Solvers](https://mlsanity.com/paper/2601.20174/) (2026)
- [Objective-Specific Privileged Bases via Full-Prefix Matryoshka Learning](https://mlsanity.com/paper/2605.09160/) (2026)
- [On the Regularization Landscape for the Linear Recommendation Models](https://mlsanity.com/paper/2609.11876/) (2026)

## Abstract (arXiv)

Our understanding of learning input-output relationships with neural nets has improved rapidly in recent years, but little is known about the convergence of the underlying representations, even in the simple case of linear autoencoders (LAEs). We show that when trained with proper regularization, LAEs can directly learn the optimal representation -- ordered, axis-aligned principal components. We analyze two such regularization schemes: non-uniform ℓ2 regularization and a deterministic variant of nested dropout [Rippel et al, ICML' 2014]. Though both regularization schemes converge to the optimal representation, we show that this convergence is slow due to ill-conditioning that worsens with increasing latent dimension. We show that the inefficiency of learning the optimal representation is not inevitable -- we present a simple modification to the gradient descent update that greatly speeds up convergence empirically.