# Unsupervised Representation Disentanglement of Text: An Evaluation on Synthetic Datasets

arXiv: https://arxiv.org/abs/2106.03631 · HTML: https://mlsanity.com/paper/2106.03631/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 45 references that are in the map.
- First posted on arXiv 7 Jun 2021.

## Builds on (most cited first, 30 of 32)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Character-level Convolutional Networks for Text Classification](https://mlsanity.com/paper/1509.01626/) (2015)
- [Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations](https://mlsanity.com/paper/1811.12359/) (2018)
- [Designing and Interpreting Probes with Control Tasks](https://mlsanity.com/paper/1909.03368/) (2019)
- [Disentangling by Factorising](https://mlsanity.com/paper/1802.05983/) (2018)
- [Generating Sentences from a Continuous Space](https://mlsanity.com/paper/1511.06349/) (2015)
- [What you can cram into a single vector: Probing sentence embeddings for linguistic properties](https://mlsanity.com/paper/1805.01070/) (2018)
- [Understanding disentangling in β-VAE](https://mlsanity.com/paper/1804.03599/) (2018)
- [Towards a Definition of Disentangled Representations](https://mlsanity.com/paper/1812.02230/) (2018)
- [Targeted Syntactic Evaluation of Language Models](https://mlsanity.com/paper/1808.09031/) (2018)
- [Variational Inference of Disentangled Latent Concepts from Unlabeled Observations](https://mlsanity.com/paper/1711.00848/) (2017)
- [Information-Theoretic Probing with Minimum Description Length](https://mlsanity.com/paper/2003.12298/) (2020)
- [Information-Theoretic Probing for Linguistic Structure](https://mlsanity.com/paper/2004.03061/) (2020)
- [Disentangled Representation Learning for Non-Parallel Text Style Transfer](https://mlsanity.com/paper/1808.04339/) (2018)
- [Lagging Inference Networks and Posterior Collapse in Variational Autoencoders](https://mlsanity.com/paper/1901.05534/) (2019)
- [Learning Deep Disentangled Embeddings with the F-Statistic Loss](https://mlsanity.com/paper/1802.05312/) (2018)
- [Preventing Posterior Collapse with delta-VAEs](https://mlsanity.com/paper/1901.03416/) (2019)
- [Deep Learning of Representations: Looking Forward](https://mlsanity.com/paper/1305.0445/) (2013)
- [Improved Variational Autoencoders for Text Modeling using Dilated Convolutions](https://mlsanity.com/paper/1702.08139/) (2017)
- [On the Transfer of Disentangled Representations in Realistic Settings](https://mlsanity.com/paper/2010.14407/) (2020)
- [Generating Sentences from Disentangled Syntactic and Semantic Spaces](https://mlsanity.com/paper/1907.05789/) (2019)
- [Improving Disentangled Text Representation Learning with Information-Theoretic Guidance](https://mlsanity.com/paper/2006.00693/) (2020)
- [Avoiding Latent Variable Collapse With Generative Skip Models](https://mlsanity.com/paper/1807.04863/) (2018)
- [Intrinsic Probing through Dimension Selection](https://mlsanity.com/paper/2010.02812/) (2020)
- [A Multi-Task Approach for Disentangling Syntax and Semantics in Sentence Representations](https://mlsanity.com/paper/1904.01173/) (2019)
- [Assessing Composition in Sentence Vector Representations](https://mlsanity.com/paper/1809.03992/) (2018)
- [Benchmarks, Algorithms, and Metrics for Hierarchical Disentanglement](https://mlsanity.com/paper/2102.05185/) (2021)
- [Effective Estimation of Deep Generative Language Models](https://mlsanity.com/paper/1904.08194/) (2019)
- [On the Importance of the Kullback-Leibler Divergence Term in Variational Autoencoders for Text Generation](https://mlsanity.com/paper/1909.13668/) (2019)
- [Polarized-VAE: Proximity Based Disentangled Representation Learning for Text Generation](https://mlsanity.com/paper/2004.10809/) (2020)

## Built on by (most cited first, 1 of 1)

- [Learning Disentangled Representations for Natural Language Definitions](https://mlsanity.com/paper/2210.02898/) (2022)

## Abstract (arXiv)

To highlight the challenges of achieving representation disentanglement for text domain in an unsupervised setting, in this paper we select a representative set of successfully applied models from the image domain. We evaluate these models on 6 disentanglement metrics, as well as on downstream classification tasks and homotopy. To facilitate the evaluation, we propose two synthetic datasets with known generative factors. Our experiments highlight the existing gap in the text domain and illustrate that certain elements such as representation sparsity (as an inductive bias), or representation coupling with the decoder could impact disentanglement. To the best of our knowledge, our work is the first attempt on the intersection of unsupervised representation disentanglement and text, and provides the experimental framework and datasets for examining future developments in this direction.