# Effective Estimation of Deep Generative Language Models

arXiv: https://arxiv.org/abs/1904.08194 · HTML: https://mlsanity.com/paper/1904.08194/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Apr 2019.

## Built on by (most cited first, 9 of 9)

- [AdaVAE: Exploring Adaptive GPT-2s in Variational Auto-Encoders for Language Modeling](https://mlsanity.com/paper/2205.05862/) (2022)
- [Generative Text Modeling through Short Run Inference](https://mlsanity.com/paper/2106.02513/) (2021)
- [PCAE: A Framework of Plug-in Conditional Auto-Encoder for Controllable Text Generation](https://mlsanity.com/paper/2210.03496/) (2022)
- [Statistical Model Criticism of Variational Auto-Encoders](https://mlsanity.com/paper/2204.03030/) (2022)
- [Preventing posterior collapse in variational autoencoders for text generation via decoder regularization](https://mlsanity.com/paper/2110.14945/) (2021)
- [Unsupervised Representation Disentanglement of Text: An Evaluation on Synthetic Datasets](https://mlsanity.com/paper/2106.03631/) (2021)
- [Fair Inference for Discrete Latent Variable Models](https://mlsanity.com/paper/2209.07044/) (2022)
- [PrOnto: Language Model Evaluations for 859 Languages](https://mlsanity.com/paper/2305.12612/) (2023)
- [The Neglected Sibling: Isotropic Gaussian Posterior for VAE](https://mlsanity.com/paper/2110.07383/) (2021)

## Abstract (arXiv)

Advances in variational inference enable parameterisation of probabilistic models by deep neural networks. This combines the statistical transparency of the probabilistic modelling framework with the representational power of deep learning. Yet, due to a problem known as posterior collapse, it is difficult to estimate such models in the context of language modelling effectively. We concentrate on one such model, the variational auto-encoder, which we argue is an important building block in hierarchical probabilistic models of language. This paper contributes a sober view of the problem, a survey of techniques to address it, novel techniques, and extensions to the model. To establish a ranking of techniques, we perform a systematic comparison using Bayesian optimisation and find that many techniques perform reasonably similar, given enough resources. Still, a favourite can be named based on convenience. We also make several empirical observations and recommendations of best practices that should help researchers interested in this exciting field.