# Implicit Rank-Minimizing Autoencoder

arXiv: https://arxiv.org/abs/2010.00679 · HTML: https://mlsanity.com/paper/2010.00679/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Oct 2020.

## Built on by (most cited first, 30 of 35)

- [Understanding Dimensional Collapse in Contrastive Self-supervised Learning](https://mlsanity.com/paper/2110.09348/) (2021)
- [The Low-Rank Simplicity Bias in Deep Networks](https://mlsanity.com/paper/2103.10427/) (2021)
- [How Mask Matters: Towards Theoretical Understandings of Masked Autoencoders](https://mlsanity.com/paper/2210.08344/) (2022)
- [Data-driven discovery of intrinsic dynamics](https://mlsanity.com/paper/2108.05928/) (2021)
- [Disentanglement via Latent Quantization](https://mlsanity.com/paper/2305.18378/) (2023)
- [Learning to Accelerate Partial Differential Equations via Latent Global Evolution](https://mlsanity.com/paper/2206.07681/) (2022)
- [Self-Validation: Early Stopping for Single-Instance Deep Generative Priors](https://mlsanity.com/paper/2110.12271/) (2021)
- [Deep Weight Factorization: Sparse Learning Through the Lens of Artificial Symmetries](https://mlsanity.com/paper/2502.02496/) (2025)
- [Autoencoders for discovering manifold dimension and coordinates in data from complex dynamical systems](https://mlsanity.com/paper/2305.01090/) (2023)
- [Tripod: Three Complementary Inductive Biases for Disentangled Representation Learning](https://mlsanity.com/paper/2404.10282/) (2024)
- [Residual Connections Harm Generative Representation Learning](https://mlsanity.com/paper/2404.10947/) (2024)
- [Preventing Dimensional Collapse in Self-Supervised Learning via Orthogonality Regularization](https://mlsanity.com/paper/2411.00392/) (2024)
- [Implicit Regularization for Group Sparsity](https://mlsanity.com/paper/2301.12540/) (2023)
- [Simpler is better: spectral regularization and up-sampling techniques for variational autoencoders](https://mlsanity.com/paper/2201.07544/) (2022)
- [Deep ReLU Networks Have Surprisingly Simple Polytopes](https://mlsanity.com/paper/2305.09145/) (2023)
- [Rank Reduction Autoencoders](https://mlsanity.com/paper/2405.13980/) (2024)
- [Implicit Regularization with Polynomial Growth in Deep Tensor Factorization](https://mlsanity.com/paper/2207.08942/) (2022)
- [Improved CNN-based Learning of Interpolation Filters for Low-Complexity Inter Prediction in Video Coding](https://mlsanity.com/paper/2106.08936/) (2021)
- [Building symmetries into data-driven manifold dynamics models for complex flows: application to two-dimensional Kolmogorov flow](https://mlsanity.com/paper/2312.10235/) (2023)
- [PCAE: Learning Ordered Representations in Latent Space for Intrinsic Dimension Estimation via Principal Component Autoencoder](https://mlsanity.com/paper/2601.19179/) (2026)
- [TRIE: An Evaluation Framework for Stochastic PDE Surrogates](https://mlsanity.com/paper/2607.00196/) (2026)
- [Implicit Greedy Rank Learning in Autoencoders via Overparameterized Linear Networks](https://mlsanity.com/paper/2107.01301/) (2021)
- [IHT-Inspired Neural Network for Single-Snapshot DOA Estimation with Sparse Linear Arrays](https://mlsanity.com/paper/2309.08429/) (2023)
- [Learning Low-Rank Latent Spaces with Simple Deterministic Autoencoder: Theoretical and Empirical Insights](https://mlsanity.com/paper/2310.16194/) (2023)
- [Lecture Notes on Linear Neural Networks: A Tale of Optimization and Generalization in Deep Learning](https://mlsanity.com/paper/2408.13767/) (2024)
- [Characterizing Continuous and Discrete Hybrid Latent Spaces for Structural Connectomes](https://mlsanity.com/paper/2512.02032/) (2025)
- [Eccentric Regularization: Minimizing Hyperspherical Energy without explicit projection](https://mlsanity.com/paper/2104.11610/) (2021)
- [A Study of Representational Properties of Unsupervised Anomaly Detection in Brain MRI](https://mlsanity.com/paper/2211.15527/) (2022)
- [Learning Linear Groups in Neural Networks](https://mlsanity.com/paper/2305.18552/) (2023)
- [A Theoretical and Empirical Study on the Convergence of Adam with an "Exact" Constant Step Size in Non-Convex Settings](https://mlsanity.com/paper/2309.08339/) (2023)

## Abstract (arXiv)

An important component of autoencoders is the method by which the information capacity of the latent representation is minimized or limited. In this work, the rank of the covariance matrix of the codes is implicitly minimized by relying on the fact that gradient descent learning in multi-layer linear networks leads to minimum-rank solutions. By inserting a number of extra linear layers between the encoder and the decoder, the system spontaneously learns representations with a low effective dimension. The model, dubbed Implicit Rank-Minimizing Autoencoder (IRMAE), is simple, deterministic, and learns compact latent spaces. We demonstrate the validity of the method on several image generation and representation learning tasks.