# Deep Directed Generative Autoencoders

arXiv: https://arxiv.org/abs/1410.0630 · HTML: https://mlsanity.com/paper/1410.0630/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Oct 2014.

## Built on by (most cited first, 2 of 2)

- [Deep Grokking: Would Deep Neural Networks Generalize Better?](https://mlsanity.com/paper/2405.19454/) (2024)
- [Student-Oriented Teacher Knowledge Refinement for Knowledge Distillation](https://mlsanity.com/paper/2409.18785/) (2024)

## Abstract (arXiv)

For discrete data, the likelihood P(x) can be rewritten exactly and parametrized into P(X = x) = P(X = x | H = f(x)) P(H = f(x)) if P(X | H) has enough capacity to put no probability mass on any x' for which f(x')neq f(x), where f(·) is a deterministic discrete function. The log of the first factor gives rise to the log-likelihood reconstruction error of an autoencoder with f(·) as the encoder and P(X|H) as the (probabilistic) decoder. The log of the second term can be seen as a regularizer on the encoded activations h=f(x), e.g., as in sparse autoencoders. Both encoder and decoder can be represented by a deep neural network and trained to maximize the average of the optimal log-likelihood log p(x). The objective is to learn an encoder f(·) that maps X to f(X) that has a much simpler distribution than X itself, estimated by P(H). This "flattens the manifold" or concentrates probability mass in a smaller number of (relevant) dimensions over which the distribution factorizes. Generating samples from the model is straightforward using ancestral sampling. One challenge is that regular back-propagation cannot be used to obtain the gradient on the parameters of the encoder, but we find that using the straight-through estimator works well here. We also find that although optimizing a single level of such architecture may be difficult, much better results can be obtained by pre-training and stacking them, gradually transforming the data distribution into one that is more easily captured by a simple parametric model.