# Topic Compositional Neural Language Model

arXiv: https://arxiv.org/abs/1712.09783 · HTML: https://mlsanity.com/paper/1712.09783/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Placed in MoE large language models, one of 12 directions in Mixture-of-experts.
- First posted on arXiv 28 Dec 2017.

## Placed in

- Mixture-of-experts / [MoE large language models](https://mlsanity.com/moe/moe-large-language-models/)

## Built on by (most cited first, 11 of 11)

- [Topic Modelling Meets Deep Neural Networks: A Survey](https://mlsanity.com/paper/2103.00498/) (2021)
- [HyperMiner: Topic Taxonomy Mining with Hyperbolic Embedding](https://mlsanity.com/paper/2210.10625/) (2022)
- [KALM: Knowledge-Aware Integration of Local, Document, and Global Contexts for Long Document Understanding](https://mlsanity.com/paper/2210.04105/) (2022)
- [Selective Text Augmentation with Word Roles for Low-Resource Text Classification](https://mlsanity.com/paper/2209.01560/) (2022)
- [Revisiting Topic-Guided Language Models](https://mlsanity.com/paper/2312.02331/) (2023)
- [Not All Comments are Equal: Insights into Comment Moderation from a Topic-Aware Model](https://mlsanity.com/paper/2109.10033/) (2021)
- [TopNet: Learning from Neural Topic Model to Generate Long Stories](https://mlsanity.com/paper/2112.07259/) (2021)
- [What Have Been Learned & What Should Be Learned? An Empirical Study of How to Selectively Augment Text for Classification](https://mlsanity.com/paper/2109.00175/) (2021)
- [An Overview on Controllable Text Generation via Variational Auto-Encoders](https://mlsanity.com/paper/2211.07954/) (2022)
- [Recurrent Coupled Topic Modeling over Sequential Documents](https://mlsanity.com/paper/2106.13732/) (2021)
- [Explainable Topic-Enhanced Argument Mining from Heterogeneous Sources](https://mlsanity.com/paper/2307.12131/) (2023)

## Abstract (arXiv)

We propose a Topic Compositional Neural Language Model (TCNLM), a novel method designed to simultaneously capture both the global semantic meaning and the local word ordering structure in a document. The TCNLM learns the global semantic coherence of a document via a neural topic model, and the probability of each learned latent topic is further used to build a Mixture-of-Experts (MoE) language model, where each expert (corresponding to one topic) is a recurrent neural network (RNN) that accounts for learning the local structure of a word sequence. In order to train the MoE model efficiently, a matrix factorization method is applied, by extending each weight matrix of the RNN to be an ensemble of topic-dependent weight matrices. The degree to which each member of the ensemble is used is tied to the document-dependent probability of the corresponding topics. Experimental results on several corpora show that the proposed approach outperforms both a pure RNN-based model and other topic-guided language models. Further, our model yields sensible topics, and also has the capacity to generate meaningful sentences conditioned on given topics.