# Sawtooth Factorial Topic Embeddings Guided Gamma Belief Network

arXiv: https://arxiv.org/abs/2107.02757 · HTML: https://mlsanity.com/paper/2107.02757/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 50 references that are in the map.
- First posted on arXiv 30 Jun 2021.

## Builds on (most cited first, 24 of 24)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [Compressive Transformers for Long-Range Sequence Modelling](https://mlsanity.com/paper/1911.05507/) (2019)
- [NVAE: A Deep Hierarchical Variational Autoencoder](https://mlsanity.com/paper/2007.03898/) (2020)
- [Ladder Variational Autoencoders](https://mlsanity.com/paper/1602.02282/) (2016)
- [Very Deep VAEs Generalize Autoregressive Models and Can Outperform Them on Images](https://mlsanity.com/paper/2011.10650/) (2020)
- [Topic Modeling in Embedding Spaces](https://mlsanity.com/paper/1907.04907/) (2019)
- [Autoencoding Variational Inference For Topic Models](https://mlsanity.com/paper/1703.01488/) (2017)
- [Neural Variational Inference for Text Processing](https://mlsanity.com/paper/1511.06038/) (2015)
- [BIVA: A Very Deep Hierarchy of Latent Variables for Generative Modeling](https://mlsanity.com/paper/1902.02102/) (2019)
- [Preventing Posterior Collapse with delta-VAEs](https://mlsanity.com/paper/1901.03416/) (2019)
- [Greedy Hierarchical Variational Autoencoders for Large-Scale Video Prediction](https://mlsanity.com/paper/2103.04174/) (2021)
- [Topic Modeling with Wasserstein Autoencoders](https://mlsanity.com/paper/1907.12374/) (2019)
- [Bayesian Attention Modules](https://mlsanity.com/paper/2010.10604/) (2020)
- [Topic Modelling Meets Deep Neural Networks: A Survey](https://mlsanity.com/paper/2103.00498/) (2021)
- [The Dynamic Embedded Topic Model](https://mlsanity.com/paper/1907.05545/) (2019)
- [WHAI: Weibull Hybrid Autoencoding Inference for Deep Topic Modeling](https://mlsanity.com/paper/1803.01328/) (2018)
- [Stochastic Backpropagation and Approximate Inference in Deep Generative Models](https://mlsanity.com/paper/1401.4082/) (2014)
- [Neural Topic Modeling with Bidirectional Adversarial Training](https://mlsanity.com/paper/2004.12331/) (2020)
- [Dirichlet belief networks for topic structure learning](https://mlsanity.com/paper/1811.00717/) (2018)
- [The Poisson Gamma Belief Network](https://mlsanity.com/paper/1511.02199/) (2015)
- [Deep Latent Dirichlet Allocation with Topic-Layer-Adaptive Stochastic Gradient Riemannian MCMC](https://mlsanity.com/paper/1706.01724/) (2017)
- [Variational Hetero-Encoder Randomized GANs for Joint Image-Text Modeling](https://mlsanity.com/paper/1905.08622/) (2019)

## Built on by (most cited first, 23 of 23)

- [Representing Mixtures of Word Embeddings with Mixtures of Topic Embeddings](https://mlsanity.com/paper/2203.01570/) (2022)
- [Towards the TopMost: A Topic Modeling System Toolkit](https://mlsanity.com/paper/2309.06908/) (2023)
- [On the Affinity, Rationality, and Diversity of Hierarchical Topic Modeling](https://mlsanity.com/paper/2401.14113/) (2024)
- [A Survey on Neural Topic Models: Methods, Applications, and Challenges](https://mlsanity.com/paper/2401.15351/) (2024)
- [vONTSS: vMF based semi-supervised neural topic modeling with optimal transport](https://mlsanity.com/paper/2307.01226/) (2023)
- [HyperMiner: Topic Taxonomy Mining with Hyperbolic Embedding](https://mlsanity.com/paper/2210.10625/) (2022)
- [Alignment Attention by Matching Key and Query Distributions](https://mlsanity.com/paper/2110.12567/) (2021)
- [Neural Topic Modeling with Large Language Models in the Loop](https://mlsanity.com/paper/2411.08534/) (2024)
- [TopicNet: Semantic Graph-Guided Topic Discovery](https://mlsanity.com/paper/2110.14286/) (2021)
- [Knowledge-Aware Bayesian Deep Topic Model](https://mlsanity.com/paper/2209.14228/) (2022)
- [Topic-VQ-VAE: Leveraging Latent Codebooks for Flexible Topic-Guided Document Generation](https://mlsanity.com/paper/2312.11532/) (2023)
- [Mitigating Reward Hacking in RLHF via Bayesian Non-negative Reward Modeling](https://mlsanity.com/paper/2602.10623/) (2026)
- [A Non-negative VAE:the Generalized Gamma Belief Network](https://mlsanity.com/paper/2408.03388/) (2024)
- [Self-supervised Topic Taxonomy Discovery in the Box Embedding Space](https://mlsanity.com/paper/2408.15050/) (2024)
- [Enhancing Uncertainty Estimation and Interpretability via Bayesian Non-negative Decision Layer](https://mlsanity.com/paper/2505.22199/) (2025)
- [Improving Sparse Autoencoder with Dynamic Attention](https://mlsanity.com/paper/2604.14925/) (2026)
- [HyHTM: Hyperbolic Geometry based Hierarchical Topic Models](https://mlsanity.com/paper/2305.09258/) (2023)
- [Disentangled Generative Graph Representation Learning](https://mlsanity.com/paper/2408.13471/) (2024)
- [Variational Language Concepts for Interpreting Foundation Language Models](https://mlsanity.com/paper/2410.03964/) (2024)
- [Hierarchical Graph Topic Modeling with Topic Tree-based Transformer](https://mlsanity.com/paper/2502.11345/) (2025)
- [Understanding Cross-Domain Adaptation in Low-Resource Topic Modeling](https://mlsanity.com/paper/2506.07453/) (2025)
- [CobwebTM: Probabilistic Concept Formation for Lifelong and Hierarchical Topic Modeling](https://mlsanity.com/paper/2604.14489/) (2026)
- [Scalable Amortized Variational Inference for Non-Poisson Buy-'Til-You-Die Models](https://mlsanity.com/paper/2608.19022/) (2026)

## Abstract (arXiv)

Hierarchical topic models such as the gamma belief network (GBN) have delivered promising results in mining multi-layer document representations and discovering interpretable topic taxonomies. However, they often assume in the prior that the topics at each layer are independently drawn from the Dirichlet distribution, ignoring the dependencies between the topics both at the same layer and across different layers. To relax this assumption, we propose sawtooth factorial topic embedding guided GBN, a deep generative model of documents that captures the dependencies and semantic similarities between the topics in the embedding space. Specifically, both the words and topics are represented as embedding vectors of the same dimension. The topic matrix at a layer is factorized into the product of a factor loading matrix and a topic embedding matrix, the transpose of which is set as the factor loading matrix of the layer above. Repeating this particular type of factorization, which shares components between adjacent layers, leads to a structure referred to as sawtooth factorization. An auto-encoding variational inference network is constructed to optimize the model parameter via stochastic gradient descent. Experiments on big corpora show that our models outperform other neural topic models on extracting deeper interpretable topics and deriving better document representations.