# TopicRNN: A Recurrent Neural Network with Long-Range Semantic Dependency

arXiv: https://arxiv.org/abs/1611.01702 · HTML: https://mlsanity.com/paper/1611.01702/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Nov 2016.

## Built on by (most cited first, 27 of 27)

- [Topic Modelling Meets Deep Neural Networks: A Survey](https://mlsanity.com/paper/2103.00498/) (2021)
- [Consistency Regularization for Variational Auto-Encoders](https://mlsanity.com/paper/2105.14859/) (2021)
- [FASTopic: Pretrained Transformer is a Fast, Adaptive, Stable, and Transferable Topic Model](https://mlsanity.com/paper/2405.17978/) (2024)
- [Focus Attention: Promoting Faithfulness and Diversity in Summarization](https://mlsanity.com/paper/2105.11921/) (2021)
- [Rationalization through Concepts](https://mlsanity.com/paper/2105.04837/) (2021)
- [A Survey on Neural Topic Models: Methods, Applications, and Challenges](https://mlsanity.com/paper/2401.15351/) (2024)
- [Imitation Learning-based Implicit Semantic-aware Communication Networks: Multi-layer Representation and Collaborative Reasoning](https://mlsanity.com/paper/2210.16118/) (2022)
- [Graph Neural Networks for Text Classification: A Survey](https://mlsanity.com/paper/2304.11534/) (2023)
- [Graph Contrastive Learning via Cluster-refined Negative Sampling for Semi-supervised Text Classification](https://mlsanity.com/paper/2410.18130/) (2024)
- [GORAG: Graph-based Online Retrieval Augmented Generation for Dynamic Few-shot Social Media Text Classification](https://mlsanity.com/paper/2501.02844/) (2025)
- [Contrastive Multi-graph Learning with Neighbor Hierarchical Sifting for Semi-supervised Text Classification](https://mlsanity.com/paper/2411.16787/) (2024)
- [Deep Learning-based Sentiment Classification: A Comparative Survey](https://mlsanity.com/paper/2312.17253/) (2023)
- [PriorCLIP: Visual Prior Guided Vision-Language Model for Remote Sensing Image-Text Retrieval](https://mlsanity.com/paper/2405.10160/) (2024)
- [KDSTM: Neural Semi-supervised Topic Modeling with Knowledge Distillation](https://mlsanity.com/paper/2307.01878/) (2023)
- [Revisiting Topic-Guided Language Models](https://mlsanity.com/paper/2312.02331/) (2023)
- [Alternators For Sequence Modeling](https://mlsanity.com/paper/2405.11848/) (2024)
- [Not All Comments are Equal: Insights into Comment Moderation from a Topic-Aware Model](https://mlsanity.com/paper/2109.10033/) (2021)
- [TopNet: Learning from Neural Topic Model to Generate Long Stories](https://mlsanity.com/paper/2112.07259/) (2021)
- [Bayes-CATSI: A variational Bayesian deep learning framework for medical time series data imputation](https://mlsanity.com/paper/2410.01847/) (2024)
- [Beyond Topics: Discovering Latent Healthcare Objectives from Event Sequences](https://mlsanity.com/paper/2110.01160/) (2021)
- [C5: Towards Better Conversation Comprehension and Contextual Continuity for ChatGPT](https://mlsanity.com/paper/2308.05567/) (2023)
- [Deep Probabilistic Graphical Modeling](https://mlsanity.com/paper/2104.12053/) (2021)
- [Recurrent Coupled Topic Modeling over Sequential Documents](https://mlsanity.com/paper/2106.13732/) (2021)
- [Learning Sequence Representations by Non-local Recurrent Neural Memory](https://mlsanity.com/paper/2207.09710/) (2022)
- [Measuring Adversarial Datasets](https://mlsanity.com/paper/2311.03566/) (2023)
- [Extrapolated Markov Chain Oversampling Method for Imbalanced Text Classification](https://mlsanity.com/paper/2509.02332/) (2025)
- [Idea-Gated Transformers: Enforcing Semantic Coherence via Differentiable Vocabulary Pruning](https://mlsanity.com/paper/2512.03343/) (2025)

## Abstract (arXiv)

In this paper, we propose TopicRNN, a recurrent neural network (RNN)-based language model designed to directly capture the global semantic meaning relating words in a document via latent topics. Because of their sequential nature, RNNs are good at capturing the local structure of a word sequence - both semantic and syntactic - but might face difficulty remembering long-range dependencies. Intuitively, these long-range dependencies are of semantic nature. In contrast, latent topic models are able to capture the global underlying semantic structure of a document but do not account for word ordering. The proposed TopicRNN model integrates the merits of RNNs and latent topic models: it captures local (syntactic) dependencies using an RNN and global (semantic) dependencies using latent topics. Unlike previous work on contextual RNN language modeling, our model is learned end-to-end. Empirical results on word prediction show that TopicRNN outperforms existing contextual RNN baselines. In addition, TopicRNN can be used as an unsupervised feature extractor for documents. We do this for sentiment analysis on the IMDB movie review dataset and report an error rate of 6.28%. This is comparable to the state-of-the-art 5.91% resulting from a semi-supervised approach. Finally, TopicRNN also yields sensible topics, making it a useful alternative to document models such as latent Dirichlet allocation.