# Improving Disentangled Text Representation Learning with Information-Theoretic Guidance

arXiv: https://arxiv.org/abs/2006.00693 · HTML: https://mlsanity.com/paper/2006.00693/

## Key facts

- Cited by 52 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Jun 2020.

## Built on by (most cited first, 30 of 52)

- [Disentangled Representation Learning](https://mlsanity.com/paper/2211.11695/) (2022)
- [FairFil: Contrastive Neural Debiasing Method for Pretrained Text Encoders](https://mlsanity.com/paper/2103.06413/) (2021)
- [A Novel Estimator of Mutual Information for Learning to Disentangle Textual Representations](https://mlsanity.com/paper/2105.02685/) (2021)
- [Contrastively Disentangled Sequential Variational Autoencoder](https://mlsanity.com/paper/2110.12091/) (2021)
- [Disentangled Sequence to Sequence Learning for Compositional Generalization](https://mlsanity.com/paper/2110.04655/) (2021)
- [Black-box Adversarial Attacks against Dense Retrieval Models: A Multi-view Contrastive Learning Method](https://mlsanity.com/paper/2308.09861/) (2023)
- [Parameter-efficient Modularised Bias Mitigation via AdapterFusion](https://mlsanity.com/paper/2302.06321/) (2023)
- [A Differential Entropy Estimator for Training Neural Networks](https://mlsanity.com/paper/2202.06618/) (2022)
- [Disentangling Generative Factors in Natural Language with Discrete Variational Autoencoders](https://mlsanity.com/paper/2109.07169/) (2021)
- [Modular and On-demand Bias Mitigation with Attribute-Removal Subnetworks](https://mlsanity.com/paper/2205.15171/) (2022)
- [Unsupervised Structure-Texture Separation Network for Oracle Character Recognition](https://mlsanity.com/paper/2205.06549/) (2022)
- [Adversarially learning disentangled speech representations for robust multi-factor voice conversion](https://mlsanity.com/paper/2102.00184/) (2021)
- [Learning Disentangled Textual Representations via Statistical Measures of Similarity](https://mlsanity.com/paper/2205.03589/) (2022)
- [Learning Disentangled Representations of Negation and Uncertainty](https://mlsanity.com/paper/2204.00511/) (2022)
- [Learning Disentangled Representations for Natural Language Definitions](https://mlsanity.com/paper/2210.02898/) (2022)
- [Conditional Supervised Contrastive Learning for Fair Text Classification](https://mlsanity.com/paper/2205.11485/) (2022)
- [DIFFER: Disentangling Identity Features via Semantic Cues for Clothes-Changing Person Re-ID](https://mlsanity.com/paper/2503.22912/) (2025)
- [TexShape: Information Theoretic Sentence Embedding for Language Models](https://mlsanity.com/paper/2402.05132/) (2024)
- [Graph-based Unsupervised Disentangled Representation Learning via Multimodal Large Language Models](https://mlsanity.com/paper/2407.18999/) (2024)
- [Closed-Loop Unsupervised Representation Disentanglement with β-VAE Distillation and Diffusion Probabilistic Feedback](https://mlsanity.com/paper/2402.02346/) (2024)
- [Contrastive Representation Learning for Exemplar-Guided Paraphrase Generation](https://mlsanity.com/paper/2109.01484/) (2021)
- [From Theories on Styles to their Transfer in Text: Bridging the Gap with a Hierarchical Survey](https://mlsanity.com/paper/2110.15871/) (2021)
- [Towards Unsupervised Content Disentanglement in Sentence Representations via Syntactic Roles](https://mlsanity.com/paper/2206.11184/) (2022)
- [Disentangled Variational Autoencoder for Emotion Recognition in Conversations](https://mlsanity.com/paper/2305.14071/) (2023)
- [Preventing Author Profiling through Zero-Shot Multilingual Back-Translation](https://mlsanity.com/paper/2109.09133/) (2021)
- [Disentangled Text Representation Learning with Information-Theoretic Perspective for Adversarial Robustness](https://mlsanity.com/paper/2210.14957/) (2022)
- [Enhanced Coherence-Aware Network with Hierarchical Disentanglement for Aspect-Category Sentiment Analysis](https://mlsanity.com/paper/2403.10214/) (2024)
- [Exploiting Inductive Bias in Transformers for Unsupervised Disentanglement of Syntax and Semantics with VAEs](https://mlsanity.com/paper/2205.05943/) (2022)
- [StyLEx: Explaining Style Using Human Lexical Annotations](https://mlsanity.com/paper/2210.07469/) (2022)
- [Replacing Language Model for Style Transfer](https://mlsanity.com/paper/2211.07343/) (2022)

## Abstract (arXiv)

Learning disentangled representations of natural language is essential for many NLP tasks, e.g., conditional text generation, style transfer, personalized dialogue systems, etc. Similar problems have been studied extensively for other forms of data, such as images and videos. However, the discrete nature of natural language makes the disentangling of textual representations more challenging (e.g., the manipulation over the data space cannot be easily achieved). Inspired by information theory, we propose a novel method that effectively manifests disentangled representations of text, without any supervision on semantics. A new mutual information upper bound is derived and leveraged to measure dependence between style and content. By minimizing this upper bound, the proposed method induces style and content embeddings into two independent low-dimensional spaces. Experiments on both conditional text generation and text-style transfer demonstrate the high quality of our disentangled representation in terms of content and style preservation.