# Deep Discrete Encoders: Identifiable Deep Generative Models for Rich Data with Discrete Latent Layers

arXiv: https://arxiv.org/abs/2501.01414 · HTML: https://mlsanity.com/paper/2501.01414/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 53 references that are in the map.
- First posted on arXiv 2 Jan 2025.

## Builds on (most cited first, 15 of 15)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Variational Autoencoders and Nonlinear ICA: A Unifying Framework](https://mlsanity.com/paper/1907.04809/) (2019)
- [Nonlinear ICA Using Auxiliary Variables and Generalized Contrastive Learning](https://mlsanity.com/paper/1805.08651/) (2018)
- [Identifiability of deep generative models without auxiliary information](https://mlsanity.com/paper/2206.10044/) (2022)
- [Identifiable Deep Generative Models via Sparse Decoding](https://mlsanity.com/paper/2110.10804/) (2021)
- [Deep Transformation-Invariant Clustering](https://mlsanity.com/paper/2006.11132/) (2020)
- [Vintage Factor Analysis with Varimax Performs Statistical Inference](https://mlsanity.com/paper/2004.05387/) (2020)
- [Learning Discrete Concepts in Latent Hierarchical Models](https://mlsanity.com/paper/2406.00519/) (2024)
- [Bayesian Pyramids: Identifiable Multilayer Discrete Latent Structure Models for Discrete Data](https://mlsanity.com/paper/2101.10373/) (2021)
- [Deep Exponential Families](https://mlsanity.com/paper/1411.2581/) (2014)
- [Generalized Independent Noise Condition for Estimating Causal Structure with Latent Variables](https://mlsanity.com/paper/2308.06718/) (2023)
- [Modeling Documents with Deep Boltzmann Machines](https://mlsanity.com/paper/1309.6865/) (2013)
- [A Note on Exploratory Item Factor Analysis by Singular Value Decomposition](https://mlsanity.com/paper/1907.08713/) (2019)
- [Learning Topic Models: Identifiability and Finite-Sample Analysis](https://mlsanity.com/paper/2110.04232/) (2021)

## Built on by (most cited first, 2 of 2)

- [Unsupervised Causal Abstractions Discovery](https://mlsanity.com/paper/2606.19594/) (2026)
- [On Theoretical Identifiability of Discrete Latent Causal Graphical Models](https://mlsanity.com/paper/2505.18410/) (2025)

## Abstract (arXiv)

In the era of generative AI, deep generative models (DGMs) with latent representations have gained tremendous popularity. Despite their impressive empirical performance, the statistical properties of these models remain underexplored. DGMs are often overparametrized, non-identifiable, and uninterpretable black boxes, raising serious concerns when deploying them in high-stakes applications. Motivated by this, we propose interpretable deep generative models for rich data types with discrete latent layers, called Deep Discrete Encoders (DDEs). A DDE is a directed graphical model with multiple binary latent layers. Theoretically, we propose transparent identifiability conditions for DDEs, which imply progressively smaller sizes of the latent layers as they go deeper. Identifiability ensures consistent parameter estimation and inspires an interpretable design of the deep architecture. Computationally, we propose a scalable estimation pipeline of a layerwise nonlinear spectral initialization followed by a penalized stochastic approximation EM algorithm. This procedure can efficiently estimate models with exponentially many latent components. Extensive simulation studies for high-dimensional data and deep architectures validate our theoretical results and demonstrate the excellent performance of our algorithms. We apply DDEs to three diverse real datasets with different data types to perform hierarchical topic modeling, image representation learning, and response time modeling in educational testing.