mlsanityData to 14 Oct 2026

A Mechanism for Producing Aligned Latent Spaces with Autoencoders

arXiv 2106.15456 ↗First posted 29 Jun 2021cs.LG cs.AIcited by 5

Not on the map. Why?

Abstract (arXiv)

Aligned latent spaces, where meaningful semantic shifts in the input space correspond to a translation in the embedding space, play an important role in the success of downstream tasks such as unsupervised clustering and data imputation. In this work, we prove that linear and nonlinear autoencoders produce aligned latent spaces by stretching along the left singular vectors of the data. We fully characterize the amount of stretching in linear autoencoders and provide an initialization scheme to arbitrarily stretch along the top directions using these networks. We also quantify the amount of stretching in nonlinear autoencoders in a simplified setting. We use our theoretical results to align drug signatures across cell types in gene expression space and semantic shifts in word embedding spaces.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 20 of its 36 references that are in the map. First posted on arXiv 29 Jun 2021.

Builds on

20 of 36 references.

References · 20

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  2. 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 927
  3. 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 763
  4. 2019Deep Double Descent: Where Bigger Models and More Data Hurtcited by 672
  5. 2015Empirical Evaluation of Rectified Activations in Convolutional Networkcited by 519
  6. 2018On the Optimization of Deep Networks: Implicit Acceleration by Overparameterizationcited by 284
  7. 2019Topic Modeling in Embedding Spacescited by 224
  8. 2017All-but-the-Top: Simple and Effective Postprocessing for Word Representationscited by 212
  9. 2019Two models of double descent for weak featurescited by 195
  10. 2020Feature Learning in Infinite-Width Neural Networkscited by 190
  11. 2018A mathematical theory of semantic development in deep neural networkscited by 186
  12. 2018Algorithmic Regularization in Learning Deep Homogeneous Models: Layers are Automatically Balancedcited by 177
8 more
  1. 2019Implicit Regularization of Discrete Gradient Dynamics in Linear Neural Networkscited by 124
  2. 2018ClusterGAN : Latent Space Clustering in Generative Adversarial Networkscited by 75
  3. 2014word2vec Parameter Learning Explainedcited by 53
  4. 2019Overparameterized Neural Networks Implement Associative Memorycited by 45
  5. 2018From Principal Subspaces to Principal Components with Linear Autoencoderscited by 39
  6. 2019Identity Crisis: Memorization and Generalization under Extreme Overparameterizationcited by 33
  7. 2019Structuring Autoencoderscited by 11
  8. 2017Learning Word Embeddings from Speechcited by 3

Built on by

Cited by year

Citing papers per year: 2022: 2; 2023: 1; 2024: 0; 2025: 2.

5 papers.

  1. 2023High-dimensional Asymptotics of Denoising Autoencoderscited by 19
  2. 2025LAMP: Data-Efficient Linear Affine Weight-Space Models for Parameter-Controlled 3D Shape Generation and Extrapolationcited by 4
  3. 2022Fundamental Limits of Two-layer Autoencoders, and Achieving Them with Gradient Methodscited by 3
  4. 2022Realization of Causal Representation Learning to Adjust Confounding Bias in Latent Spacecited by 0
  5. 2025Optimal Linear Baseline Models for Scientific Machine Learningcited by 0