# LangVAE and LangSpace: Building and Probing for Language Model VAEs

arXiv: https://arxiv.org/abs/2505.00004 · HTML: https://mlsanity.com/paper/2505.00004/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 40 references that are in the map.
- First posted on arXiv 29 Mar 2025.

## Builds on (most cited first, 25 of 25)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Disentangling by Factorising](https://mlsanity.com/paper/1802.05983/) (2018)
- [Generating Sentences from a Continuous Space](https://mlsanity.com/paper/1511.06349/) (2015)
- [Hyperspherical Variational Auto-Encoders](https://mlsanity.com/paper/1804.00891/) (2018)
- [Simple BERT Models for Relation Extraction and Semantic Role Labeling](https://mlsanity.com/paper/1904.05255/) (2019)
- [Explaining Answers with Entailment Trees](https://mlsanity.com/paper/2104.08661/) (2021)
- [Optimus: Organizing Sentences via Pre-trained Modeling of a Latent Space](https://mlsanity.com/paper/2004.04092/) (2020)
- [Large Concept Models: Language Modeling in a Sentence Representation Space](https://mlsanity.com/paper/2412.08821/) (2024)
- [Generating Sentences from Disentangled Syntactic and Semantic Spaces](https://mlsanity.com/paper/1907.05789/) (2019)
- [Event knowledge in large language models: the gap between the impossible and the unlikely](https://mlsanity.com/paper/2212.01488/) (2022)
- [A Distributional Lens for Multi-Aspect Controllable Text Generation](https://mlsanity.com/paper/2210.02889/) (2022)
- [Composable Text Controls in Latent Space with ODEs](https://mlsanity.com/paper/2208.00638/) (2022)
- [Controllable Text Generation via Probability Density Estimation in the Latent Space](https://mlsanity.com/paper/2212.08307/) (2022)
- [Pythae: Unifying Generative Autoencoders in Python -- A Benchmarking Use Case](https://mlsanity.com/paper/2206.08309/) (2022)
- [Interpretability Illusions in the Generalization of Simplified Models](https://mlsanity.com/paper/2312.03656/) (2023)
- [Disentangling Generative Factors in Natural Language with Discrete Variational Autoencoders](https://mlsanity.com/paper/2109.07169/) (2021)
- [Hyperbolic VAE via Latent Gaussian Distributions](https://mlsanity.com/paper/2209.15217/) (2022)
- [Learning Disentangled Representations of Negation and Uncertainty](https://mlsanity.com/paper/2204.00511/) (2022)
- [Learning Disentangled Representations for Natural Language Definitions](https://mlsanity.com/paper/2210.02898/) (2022)
- [AdaVAE: Exploring Adaptive GPT-2s in Variational Auto-Encoders for Language Modeling](https://mlsanity.com/paper/2205.05862/) (2022)
- [Graph-Induced Syntactic-Semantic Spaces in Transformer-Based Variational AutoEncoders](https://mlsanity.com/paper/2311.08579/) (2023)
- [Improving Semantic Control in Discrete Latent Spaces with Transformer Quantized Variational Autoencoders](https://mlsanity.com/paper/2402.00723/) (2024)
- [Towards Unsupervised Content Disentanglement in Sentence Representations via Syntactic Roles](https://mlsanity.com/paper/2206.11184/) (2022)
- [INTERACTION: A Generative XAI Framework for Natural Language Inference Explanations](https://mlsanity.com/paper/2209.01061/) (2022)
- [LlaMaVAE: Guiding Large Language Model Generation via Continuous Latent Sentence Spaces](https://mlsanity.com/paper/2312.13208/) (2023)

## Built on by (most cited first, 3 of 3)

- [Debiasing Reward Models by Representation Learning with Guarantees](https://mlsanity.com/paper/2510.23751/) (2025)
- [Bridging Compositional and Distributional Semantics: A Survey on Latent Semantic Geometry via AutoEncoder](https://mlsanity.com/paper/2506.20083/) (2025)
- [Recursive Concept Evolution for Compositional Reasoning in Large Language Models](https://mlsanity.com/paper/2602.15725/) (2026)

## Abstract (arXiv)

We present LangVAE, a novel framework for modular construction of variational autoencoders (VAEs) on top of pre-trained large language models (LLMs). Such language model VAEs can encode the knowledge of their pre-trained components into more compact and semantically disentangled representations. The representations obtained in this way can be analysed with the LangVAE companion framework: LangSpace, which implements a collection of probing methods, such as vector traversal and interpolation, disentanglement measures, and cluster visualisations. LangVAE and LangSpace offer a flexible, efficient and scalable way of building and analysing textual representations, with simple integration for models available on the HuggingFace Hub. Additionally, we conducted a set of experiments with different encoder and decoder combinations, as well as annotated inputs, revealing a wide range of interactions across architectural families and sizes w.r.t. generalisation and disentanglement. Our findings demonstrate a promising framework for systematising the experimentation and understanding of textual representations.