# Improving Neural Topic Models using Knowledge Distillation

arXiv: https://arxiv.org/abs/2010.02377 · HTML: https://mlsanity.com/paper/2010.02377/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Oct 2020.

## Built on by (most cited first, 25 of 25)

- [Is Automated Topic Model Evaluation Broken?: The Incoherence of Coherence](https://mlsanity.com/paper/2107.02173/) (2021)
- [Topic Modelling Meets Deep Neural Networks: A Survey](https://mlsanity.com/paper/2103.00498/) (2021)
- [Contrastive Learning for Neural Topic Model](https://mlsanity.com/paper/2110.12764/) (2021)
- [Topic Discovery via Latent Space Clustering of Pretrained Language Model Representations](https://mlsanity.com/paper/2202.04582/) (2022)
- [Prompting Large Language Models for Topic Modeling](https://mlsanity.com/paper/2312.09693/) (2023)
- [A Survey on Neural Topic Models: Methods, Applications, and Challenges](https://mlsanity.com/paper/2401.15351/) (2024)
- [Topic Modeling as Multi-Objective Contrastive Optimization](https://mlsanity.com/paper/2402.07577/) (2024)
- [Are Neural Topic Models Broken?](https://mlsanity.com/paper/2210.16162/) (2022)
- [Deep Transfer Learning & Beyond: Transformer Language Models in Information Systems Research](https://mlsanity.com/paper/2110.08975/) (2021)
- [CWTM: Leveraging Contextualized Word Embeddings from BERT for Neural Topic Modeling](https://mlsanity.com/paper/2305.09329/) (2023)
- [Neural Attention-Aware Hierarchical Topic Model](https://mlsanity.com/paper/2110.07161/) (2021)
- [Multilingual and Multimodal Topic Modelling with Pretrained Embeddings](https://mlsanity.com/paper/2211.08057/) (2022)
- [Knowledge-Aware Bayesian Deep Topic Model](https://mlsanity.com/paper/2209.14228/) (2022)
- [Diversity-Aware Coherence Loss for Improving Neural Topic Models](https://mlsanity.com/paper/2305.16199/) (2023)
- [KDSTM: Neural Semi-supervised Topic Modeling with Knowledge Distillation](https://mlsanity.com/paper/2307.01878/) (2023)
- [XTRA: Cross-Lingual Topic Modeling with Topic and Representation Alignments](https://mlsanity.com/paper/2510.02788/) (2025)
- [Topic Modeling with Fine-tuning LLMs and Bag of Sentences](https://mlsanity.com/paper/2408.03099/) (2024)
- [Beyond Topics: Discovering Latent Healthcare Objectives from Event Sequences](https://mlsanity.com/paper/2110.01160/) (2021)
- [Survey of Generative Methods for Social Media Analysis](https://mlsanity.com/paper/2112.07041/) (2021)
- [Efficient and Flexible Topic Modeling using Pretrained Embeddings and Bag of Sentences](https://mlsanity.com/paper/2302.03106/) (2023)
- [Graph Contrastive Topic Model](https://mlsanity.com/paper/2307.02078/) (2023)
- [FATRER: Full-Attention Topic Regularizer for Accurate and Robust Conversational Emotion Recognition](https://mlsanity.com/paper/2307.12221/) (2023)
- [PRISM: LLM-Guided Semantic Clustering for High-Precision Topics](https://mlsanity.com/paper/2604.03180/) (2026)
- [LLM-XTM: Enhancing Cross-Lingual Topic Models with Large Language Models](https://mlsanity.com/paper/2605.03299/) (2026)
- [Self-Supervised Learning for Neural Topic Models with Variance-Invariance-Covariance Regularization](https://mlsanity.com/paper/2502.09944/) (2025)

## Abstract (arXiv)

Topic models are often used to identify human-interpretable topics to help make sense of large document collections. We use knowledge distillation to combine the best attributes of probabilistic topic models and pretrained transformers. Our modular method can be straightforwardly applied with any neural topic model to improve topic quality, which we demonstrate using two models having disparate architectures, obtaining state-of-the-art topic coherence. We show that our adaptable framework not only improves performance in the aggregate over all estimated topics, as is commonly reported, but also in head-to-head comparisons of aligned topics.