# Spherical Latent Spaces for Stable Variational Autoencoders

arXiv: https://arxiv.org/abs/1808.10805 · HTML: https://mlsanity.com/paper/1808.10805/

## Key facts

- Cited by 67 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 31 Aug 2018.

## Built on by (most cited first, 30 of 67)

- [CALM: Conditional Adversarial Latent Models for Directable Virtual Characters](https://mlsanity.com/paper/2305.02195/) (2023)
- [Latent Diffusion Model without Variational Autoencoder](https://mlsanity.com/paper/2510.15301/) (2025)
- [nGPT: Normalized Transformer with Representation Learning on the Hypersphere](https://mlsanity.com/paper/2410.01131/) (2024)
- [Hyperspherical Normalization for Scalable Deep Reinforcement Learning](https://mlsanity.com/paper/2502.15280/) (2025)
- [Spherical Sliced-Wasserstein](https://mlsanity.com/paper/2206.08780/) (2022)
- [Pseudo-Riemannian Graph Convolutional Networks](https://mlsanity.com/paper/2106.03134/) (2021)
- [Geodesic Multi-Modal Mixup for Robust Fine-Tuning](https://mlsanity.com/paper/2203.03897/) (2022)
- [Autoencoding Under Normalization Constraints](https://mlsanity.com/paper/2105.05735/) (2021)
- [AI-Generated Content (AIGC) for Various Data Modalities: A Survey](https://mlsanity.com/paper/2308.14177/) (2023)
- [Latent Thought Models with Variational Bayes Inference-Time Computation](https://mlsanity.com/paper/2502.01567/) (2025)
- [Symmetric Spaces for Graph Embeddings: A Finsler-Riemannian Approach](https://mlsanity.com/paper/2106.04941/) (2021)
- [MaGNET: Uniform Sampling from Deep Generative Network Manifolds Without Retraining](https://mlsanity.com/paper/2110.08009/) (2021)
- [Language Modeling with Hyperspherical Flows](https://mlsanity.com/paper/2605.11125/) (2026)
- [Hyperbolic Geometric Latent Diffusion Model for Graph Generation](https://mlsanity.com/paper/2405.03188/) (2024)
- [Energy-Based Models for Anomaly Detection: A Manifold Diffusion Recovery Approach](https://mlsanity.com/paper/2310.18677/) (2023)
- [VNE: An Effective Method for Improving Deep Representation by Manipulating Eigenvalue Distribution](https://mlsanity.com/paper/2304.01434/) (2023)
- [Hyperspherical Consistency Regularization](https://mlsanity.com/paper/2206.00845/) (2022)
- [vONTSS: vMF based semi-supervised neural topic modeling with optimal transport](https://mlsanity.com/paper/2307.01226/) (2023)
- [Preventing Oversmoothing in VAE via Generalized Variance Parameterization](https://mlsanity.com/paper/2102.08663/) (2021)
- [Hyperbolic VAE via Latent Gaussian Distributions](https://mlsanity.com/paper/2209.15217/) (2022)
- [Stereographic Spherical Sliced Wasserstein Distances](https://mlsanity.com/paper/2402.02345/) (2024)
- [SVG-T2I: Scaling Up Text-to-Image Latent Diffusion Model Without Variational Autoencoder](https://mlsanity.com/paper/2512.11749/) (2025)
- [Distance-based Hyperspherical Classification for Multi-source Open-Set Domain Adaptation](https://mlsanity.com/paper/2107.02067/) (2021)
- [Curvature Graph Generative Adversarial Networks](https://mlsanity.com/paper/2203.01604/) (2022)
- [Generating Multivariate Load States Using a Conditional Variational Autoencoder](https://mlsanity.com/paper/2110.11435/) (2021)
- [Image Generation with a Sphere Encoder](https://mlsanity.com/paper/2602.15030/) (2026)
- [TranSHER: Translating Knowledge Graph Embedding with Hyper-Ellipsoidal Restriction](https://mlsanity.com/paper/2204.13221/) (2022)
- [Beyond Contrastive Learning: A Variational Generative Model for Multilingual Retrieval](https://mlsanity.com/paper/2212.10726/) (2022)
- [SiBBlInGS: Similarity-driven Building-Block Inference using Graphs across States](https://mlsanity.com/paper/2306.04817/) (2023)
- [Hyperspherically Regularized Networks for Self-Supervision](https://mlsanity.com/paper/2105.00925/) (2021)

## Abstract (arXiv)

A hallmark of variational autoencoders (VAEs) for text processing is their combination of powerful encoder-decoder models, such as LSTMs, with simple latent distributions, typically multivariate Gaussians. These models pose a difficult optimization problem: there is an especially bad local optimum where the variational posterior always equals the prior and the model does not use the latent variable at all, a kind of "collapse" which is encouraged by the KL divergence term of the objective. In this work, we experiment with another choice of latent distribution, namely the von Mises-Fisher (vMF) distribution, which places mass on the surface of the unit hypersphere. With this choice of prior and posterior, the KL divergence term now only depends on the variance of the vMF distribution, giving us the ability to treat it as a fixed hyperparameter. We show that doing so not only averts the KL collapse, but consistently gives better likelihoods than Gaussians across a range of modeling conditions, including recurrent language modeling and bag-of-words document modeling. An analysis of the properties of our vMF representations shows that they learn richer and more nuanced structures in their latent representations than their Gaussian counterparts.