# Riemannian Normalizing Flow on Variational Wasserstein Autoencoder for Text Modeling

arXiv: https://arxiv.org/abs/1904.02399 · HTML: https://mlsanity.com/paper/1904.02399/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Apr 2019.

## Built on by (most cited first, 8 of 8)

- [normflows: A PyTorch Package for Normalizing Flows](https://mlsanity.com/paper/2302.12014/) (2023)
- [Partial Counterfactual Identification of Continuous Outcomes with a Curvature Sensitivity Model](https://mlsanity.com/paper/2306.01424/) (2023)
- [Learning Disentangled Representations for Time Series](https://mlsanity.com/paper/2105.08179/) (2021)
- [Fusing Multimodal Signals on Hyper-complex Space for Extreme Abstractive Text Summarization (TL;DR) of Scientific Contents](https://mlsanity.com/paper/2306.13968/) (2023)
- [Continuous Conditional Generative Adversarial Networks (cGAN) with Generator Regularization](https://mlsanity.com/paper/2103.14884/) (2021)
- [Optimal Transport Posterior Alignment for Cross-lingual Semantic Parsing](https://mlsanity.com/paper/2307.04096/) (2023)
- [Operator Autoencoders: Learning Physical Operations on Encoded Molecular Graphs](https://mlsanity.com/paper/2105.12295/) (2021)
- [A Robust Autoencoder Ensemble-Based Approach for Anomaly Detection in Text](https://mlsanity.com/paper/2405.13031/) (2024)

## Abstract (arXiv)

Recurrent Variational Autoencoder has been widely used for language modeling and text generation tasks. These models often face a difficult optimization problem, also known as the Kullback-Leibler (KL) term vanishing issue, where the posterior easily collapses to the prior, and the model will ignore latent codes in generative tasks. To address this problem, we introduce an improved Wasserstein Variational Autoencoder (WAE) with Riemannian Normalizing Flow (RNF) for text modeling. The RNF transforms a latent variable into a space that respects the geometric characteristics of input space, which makes posterior impossible to collapse to the non-informative prior. The Wasserstein objective minimizes the distance between the marginal distribution and the prior directly and therefore does not force the posterior to match the prior. Empirical experiments show that our model avoids KL vanishing over a range of datasets and has better performances in tasks such as language modeling, likelihood approximation, and text generation. Through a series of experiments and analysis over latent space, we show that our model learns latent distributions that respect latent space geometry and is able to generate sentences that are more diverse.