# Tired of Topic Models? Clusters of Pretrained Word Embeddings Make for Fast and Good Topics too!

arXiv: https://arxiv.org/abs/2004.14914 · HTML: https://mlsanity.com/paper/2004.14914/

## Key facts

- Cited by 73 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Apr 2020.

## Built on by (most cited first, 30 of 73)

- [BERTopic: Neural topic modeling with a class-based TF-IDF procedure](https://mlsanity.com/paper/2203.05794/) (2022)
- [How Do Transformers Learn Topic Structure: Towards a Mechanistic Understanding](https://mlsanity.com/paper/2303.04245/) (2023)
- [Superbizarre Is Not Superb: Derivational Morphology Improves BERT's Interpretation of Complex Words](https://mlsanity.com/paper/2101.00403/) (2021)
- [Effective Neural Topic Modeling with Embedding Clustering Regularization](https://mlsanity.com/paper/2306.04217/) (2023)
- [Is Neural Topic Modelling Better than Clustering? An Empirical Study on Clustering with Contextual Embeddings for Topics](https://mlsanity.com/paper/2204.09874/) (2022)
- [FASTopic: Pretrained Transformer is a Fast, Adaptive, Stable, and Transferable Topic Model](https://mlsanity.com/paper/2405.17978/) (2024)
- [Topic Discovery via Latent Space Clustering of Pretrained Language Model Representations](https://mlsanity.com/paper/2202.04582/) (2022)
- [Prompting Large Language Models for Topic Modeling](https://mlsanity.com/paper/2312.09693/) (2023)
- [Heterformer: Transformer-based Deep Node Representation Learning on Heterogeneous Text-Rich Networks](https://mlsanity.com/paper/2205.10282/) (2022)
- [A Survey on Neural Topic Models: Methods, Applications, and Challenges](https://mlsanity.com/paper/2401.15351/) (2024)
- [Event-Driven News Stream Clustering using Entity-Aware Contextual Embeddings](https://mlsanity.com/paper/2101.11059/) (2021)
- [Effective Seed-Guided Topic Discovery by Integrating Multiple Types of Contexts](https://mlsanity.com/paper/2212.06002/) (2022)
- [LLM Reading Tea Leaves: Automatically Evaluating Topic Models with Large Language Models](https://mlsanity.com/paper/2406.09008/) (2024)
- [Diverse Distributions of Self-Supervised Tasks for Meta-Learning in NLP](https://mlsanity.com/paper/2111.01322/) (2021)
- [Challenges and Opportunities in Information Manipulation Detection: An Examination of Wartime Russian Media](https://mlsanity.com/paper/2205.12382/) (2022)
- [Neural Topic Modeling with Large Language Models in the Loop](https://mlsanity.com/paper/2411.08534/) (2024)
- [Voices in a Crowd: Searching for Clusters of Unique Perspectives](https://mlsanity.com/paper/2407.14259/) (2024)
- [NeuroMax: Enhancing Neural Topic Modeling via Maximizing Mutual Information and Group Topic Regularization](https://mlsanity.com/paper/2409.19749/) (2024)
- [Deep Transfer Learning & Beyond: Transformer Language Models in Information Systems Research](https://mlsanity.com/paper/2110.08975/) (2021)
- [Seed-Guided Topic Discovery with Out-of-Vocabulary Seeds](https://mlsanity.com/paper/2205.01845/) (2022)
- [Deep de Finetti: Recovering Topic Distributions from Large Language Models](https://mlsanity.com/paper/2312.14226/) (2023)
- [GPTopic: Dynamic and Interactive Topic Representations](https://mlsanity.com/paper/2403.03628/) (2024)
- [Artificial intelligence for topic modelling in Hindu philosophy: mapping themes between the Upanishads and the Bhagavad Gita](https://mlsanity.com/paper/2205.11020/) (2022)
- [Learning to Collide: Recommendation System Model Compression with Learned Hash Functions](https://mlsanity.com/paper/2203.15837/) (2022)
- [Unsupervised Story Discovery from Continuous News Streams via Scalable Thematic Embedding](https://mlsanity.com/paper/2304.04099/) (2023)
- [Topic-VQ-VAE: Leveraging Latent Codebooks for Flexible Topic-Guided Document Generation](https://mlsanity.com/paper/2312.11532/) (2023)
- [Forecasting Credit Ratings: A Case Study where Traditional Methods Outperform Generative LLMs](https://mlsanity.com/paper/2407.17624/) (2024)
- [Searching for Structure in Unfalsifiable Claims](https://mlsanity.com/paper/2209.00495/) (2022)
- [Deep learning for COVID-19 topic modelling via Twitter: Alpha, Delta and Omicron](https://mlsanity.com/paper/2303.00135/) (2023)
- [Evaluating OpenAI's Whisper ASR for Punctuation Prediction and Topic Modeling of life histories of the Museum of the Person](https://mlsanity.com/paper/2305.14580/) (2023)

## Abstract (arXiv)

Topic models are a useful analysis tool to uncover the underlying themes within document collections. The dominant approach is to use probabilistic topic models that posit a generative story, but in this paper we propose an alternative way to obtain topics: clustering pre-trained word embeddings while incorporating document information for weighted clustering and reranking top words. We provide benchmarks for the combination of different word embeddings and clustering algorithms, and analyse their performance under dimensionality reduction with PCA. The best performing combination for our approach performs as well as classical topic models, but with lower runtime and computational complexity.