# Loss Landscapes of Regularized Linear Autoencoders

arXiv: https://arxiv.org/abs/1901.08168 · HTML: https://mlsanity.com/paper/1901.08168/

## Key facts

- Cited by 46 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Jan 2019.

## Built on by (most cited first, 30 of 46)

- [A Geometric Analysis of Neural Collapse with Unconstrained Features](https://mlsanity.com/paper/2105.02375/) (2021)
- [Noether's Learning Dynamics: Role of Symmetry Breaking in Neural Networks](https://mlsanity.com/paper/2105.02716/) (2021)
- [Stochastic Collapse: How Gradient Noise Attracts SGD Dynamics Towards Simpler Subnetworks](https://mlsanity.com/paper/2306.04251/) (2023)
- [Disentanglement with Biological Constraints: A Theory of Functional Cell Types](https://mlsanity.com/paper/2210.01768/) (2022)
- [Analyzing Monotonic Linear Interpolation in Neural Network Loss Landscapes](https://mlsanity.com/paper/2104.11044/) (2021)
- [The dynamics of representation learning in shallow, non-linear autoencoders](https://mlsanity.com/paper/2201.02115/) (2022)
- [An SDE for Modeling SAM: Theory and Insights](https://mlsanity.com/paper/2301.08203/) (2023)
- [High-dimensional Asymptotics of Denoising Autoencoders](https://mlsanity.com/paper/2305.11041/) (2023)
- [Understanding Representation Dynamics of Diffusion Models via Low-Dimensional Modeling](https://mlsanity.com/paper/2502.05743/) (2025)
- [Analysis of feature learning in weight-tied autoencoders via the mean field lens](https://mlsanity.com/paper/2102.08373/) (2021)
- [Deep Bayesian inference for seismic imaging with tasks](https://mlsanity.com/paper/2110.04825/) (2021)
- [Identifiability of Deep Polynomial Neural Networks](https://mlsanity.com/paper/2506.17093/) (2025)
- [Anytime Sampling for Autoregressive Models via Ordered Autoencoding](https://mlsanity.com/paper/2102.11495/) (2021)
- [Non-Parametric Representation Learning with Kernels](https://mlsanity.com/paper/2309.02028/) (2023)
- [Generalization of Diffusion Models Arises with a Balanced Representation Space](https://mlsanity.com/paper/2512.20963/) (2025)
- [Learning on a Razor's Edge: Identifiability and Singularity of Polynomial Neural Networks](https://mlsanity.com/paper/2505.11846/) (2025)
- [Stabilizing Sharpness-aware Minimization Through A Simple Renormalization Strategy](https://mlsanity.com/paper/2401.07250/) (2024)
- [Stochastic Gradient Descent-Induced Drift of Representation in a Two-Layer Neural Network](https://mlsanity.com/paper/2302.02563/) (2023)
- [How and When Random Feedback Works: A Case Study of Low-Rank Matrix Factorization](https://mlsanity.com/paper/2111.08706/) (2021)
- [Compression of Structured Data with Autoencoders: Provable Benefit of Nonlinearities and Depth](https://mlsanity.com/paper/2402.05013/) (2024)
- [Neural Image Compression: Generalization, Robustness, and Spectral Biases](https://mlsanity.com/paper/2307.08657/) (2023)
- [Posterior Collapse as a Phase Transition in Variational Autoencoders](https://mlsanity.com/paper/2510.01621/) (2025)
- [Subspace-Aware Sparse Autoencoders for Effective Mechanistic Interpretability](https://mlsanity.com/paper/2606.06333/) (2026)
- [Fundamental Limits of Two-layer Autoencoders, and Achieving Them with Gradient Methods](https://mlsanity.com/paper/2212.13468/) (2022)
- [Representation Learning Dynamics of Self-Supervised Models](https://mlsanity.com/paper/2309.02011/) (2023)
- [The Persian Rug: solving toy models of superposition using large-scale symmetries](https://mlsanity.com/paper/2410.12101/) (2024)
- [Memorization-Dilation: Modeling Neural Collapse Under Label Noise](https://mlsanity.com/paper/2206.05530/) (2022)
- [Evaluating Disentanglement in Generative Models Without Knowledge of Latent Factors](https://mlsanity.com/paper/2210.01760/) (2022)
- [The Effect of SGD Batch Size on Autoencoder Learning: Sparsity, Sharpness, and Feature Learning](https://mlsanity.com/paper/2308.03215/) (2023)
- [Brain-like Flexible Visual Inference by Harnessing Feedback-Feedforward Alignment](https://mlsanity.com/paper/2310.20599/) (2023)

## Abstract (arXiv)

Autoencoders are a deep learning model for representation learning. When trained to minimize the distance between the data and its reconstruction, linear autoencoders (LAEs) learn the subspace spanned by the top principal directions but cannot learn the principal directions themselves. In this paper, we prove that L2-regularized LAEs are symmetric at all critical points and learn the principal directions as the left singular vectors of the decoder. We smoothly parameterize the critical manifold and relate the minima to the MAP estimate of probabilistic PCA. We illustrate these results empirically and consider implications for PCA algorithms, computational neuroscience, and the algebraic topology of learning.