# Neural Attention-Aware Hierarchical Topic Model

arXiv: https://arxiv.org/abs/2110.07161 · HTML: https://mlsanity.com/paper/2110.07161/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 35 references that are in the map.
- First posted on arXiv 14 Oct 2021.

## Builds on (most cited first, 21 of 21)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Generating Sentences from a Continuous Space](https://mlsanity.com/paper/1511.06349/) (2015)
- [From Softmax to Sparsemax: A Sparse Model of Attention and Multi-Label Classification](https://mlsanity.com/paper/1602.02068/) (2016)
- [Topic Modeling in Embedding Spaces](https://mlsanity.com/paper/1907.04907/) (2019)
- [Autoencoding Variational Inference For Topic Models](https://mlsanity.com/paper/1703.01488/) (2017)
- [Pre-training is a Hot Topic: Contextualized Document Embeddings Improve Topic Coherence](https://mlsanity.com/paper/2004.03974/) (2020)
- [Neural Variational Inference for Text Processing](https://mlsanity.com/paper/1511.06038/) (2015)
- [Discovering Discrete Latent Topics with Neural Variational Inference](https://mlsanity.com/paper/1706.00359/) (2017)
- [Topic Modeling with Wasserstein Autoencoders](https://mlsanity.com/paper/1907.12374/) (2019)
- [Topic Modelling Meets Deep Neural Networks: A Survey](https://mlsanity.com/paper/2103.00498/) (2021)
- [Neural Topic Model via Optimal Transport](https://mlsanity.com/paper/2008.13537/) (2020)
- [Neural Models for Documents with Metadata](https://mlsanity.com/paper/1705.09296/) (2017)
- [Coherence-Aware Neural Topic Modeling](https://mlsanity.com/paper/1809.02687/) (2018)
- [Improving Neural Topic Models using Knowledge Distillation](https://mlsanity.com/paper/2010.02377/) (2020)
- [Neural Topic Modeling by Incorporating Document Relationship Graph](https://mlsanity.com/paper/2009.13972/) (2020)
- [Sparsemax and Relaxed Wasserstein for Topic Sparsity](https://mlsanity.com/paper/1810.09079/) (2018)
- [TopicBERT for Energy Efficient Document Classification](https://mlsanity.com/paper/2010.16407/) (2020)
- [On a Topic Model for Sentences](https://mlsanity.com/paper/1606.00253/) (2016)
- [SenGen: Sentence Generating Neural Variational Topic Model](https://mlsanity.com/paper/1708.00308/) (2017)

## Built on by (most cited first, 6 of 6)

- [Is Neural Topic Modelling Better than Clustering? An Empirical Study on Clustering with Contextual Embeddings for Topics](https://mlsanity.com/paper/2204.09874/) (2022)
- [CEMTM: Contextual Embedding-based Multimodal Topic Modeling](https://mlsanity.com/paper/2509.11465/) (2025)
- [Neural Topic Modeling with Deep Mutual Information Estimation](https://mlsanity.com/paper/2203.06298/) (2022)
- [Cyber-Security Knowledge Graph Generation by Hierarchical Nonnegative Matrix Factorization](https://mlsanity.com/paper/2403.16222/) (2024)
- [Graph2topic: an opensource topic modeling framework based on sentence embedding and community detection](https://mlsanity.com/paper/2304.06653/) (2023)
- [TopicDiff: A Topic-enriched Diffusion Approach for Multimodal Conversational Emotion Detection](https://mlsanity.com/paper/2403.04789/) (2024)

## Abstract (arXiv)

Neural topic models (NTMs) apply deep neural networks to topic modelling. Despite their success, NTMs generally ignore two important aspects: (1) only document-level word count information is utilized for the training, while more fine-grained sentence-level information is ignored, and (2) external semantic knowledge regarding documents, sentences and words are not exploited for the training. To address these issues, we propose a variational autoencoder (VAE) NTM model that jointly reconstructs the sentence and document word counts using combinations of bag-of-words (BoW) topical embeddings and pre-trained semantic embeddings. The pre-trained embeddings are first transformed into a common latent topical space to align their semantics with the BoW embeddings. Our model also features hierarchical KL divergence to leverage embeddings of each document to regularize those of their sentences, thereby paying more attention to semantically relevant sentences. Both quantitative and qualitative experiments have shown the efficacy of our model in 1) lowering the reconstruction errors at both the sentence and document levels, and 2) discovering more coherent topics from real-world datasets.