# Is Neural Topic Modelling Better than Clustering? An Empirical Study on Clustering with Contextual Embeddings for Topics

arXiv: https://arxiv.org/abs/2204.09874 · HTML: https://mlsanity.com/paper/2204.09874/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 28 references that are in the map.
- First posted on arXiv 21 Apr 2022.

## Builds on (most cited first, 18 of 18)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction](https://mlsanity.com/paper/1802.03426/) (2018)
- [SimCSE: Simple Contrastive Learning of Sentence Embeddings](https://mlsanity.com/paper/2104.08821/) (2021)
- [Autoencoding Variational Inference For Topic Models](https://mlsanity.com/paper/1703.01488/) (2017)
- [Unsupervised Domain Clusters in Pretrained Language Models](https://mlsanity.com/paper/2004.02105/) (2020)
- [Pre-training is a Hot Topic: Contextualized Document Embeddings Improve Topic Coherence](https://mlsanity.com/paper/2004.03974/) (2020)
- [Neural Variational Inference for Text Processing](https://mlsanity.com/paper/1511.06038/) (2015)
- [Accelerated Hierarchical Density Clustering](https://mlsanity.com/paper/1705.07321/) (2017)
- [Tired of Topic Models? Clusters of Pretrained Word Embeddings Make for Fast and Good Topics too!](https://mlsanity.com/paper/2004.14914/) (2020)
- [Cross-lingual Contextualized Topic Models with Zero-shot Learning](https://mlsanity.com/paper/2004.07737/) (2020)
- [Topic Modeling with Wasserstein Autoencoders](https://mlsanity.com/paper/1907.12374/) (2019)
- [Topic Modelling Meets Deep Neural Networks: A Survey](https://mlsanity.com/paper/2103.00498/) (2021)
- [Topic Modeling with Contextualized Word Representation Clusters](https://mlsanity.com/paper/2010.12626/) (2020)
- [Which *BERT? A Survey Organizing Contextualized Encoders](https://mlsanity.com/paper/2010.00854/) (2020)
- [Neural Attention-Aware Hierarchical Topic Model](https://mlsanity.com/paper/2110.07161/) (2021)
- [Bibliographic Analysis with the Citation Network Topic Model](https://mlsanity.com/paper/1609.06826/) (2016)

## Built on by (most cited first, 30 of 35)

- [How Do Transformers Learn Topic Structure: Towards a Mechanistic Understanding](https://mlsanity.com/paper/2303.04245/) (2023)
- [Effective Neural Topic Modeling with Embedding Clustering Regularization](https://mlsanity.com/paper/2306.04217/) (2023)
- [FASTopic: Pretrained Transformer is a Fast, Adaptive, Stable, and Transferable Topic Model](https://mlsanity.com/paper/2405.17978/) (2024)
- [On Behalf of the Stakeholders: Trends in NLP Model Interpretability in the Era of LLMs](https://mlsanity.com/paper/2407.19200/) (2024)
- [Goodtriever: Adaptive Toxicity Mitigation with Retrieval-augmented Models](https://mlsanity.com/paper/2310.07589/) (2023)
- [A Survey on Neural Topic Models: Methods, Applications, and Challenges](https://mlsanity.com/paper/2401.15351/) (2024)
- [DeTiME: Diffusion-Enhanced Topic Modeling using Encoder-decoder based LLM](https://mlsanity.com/paper/2310.15296/) (2023)
- [Effective Seed-Guided Topic Discovery by Integrating Multiple Types of Contexts](https://mlsanity.com/paper/2212.06002/) (2022)
- [NeuroMax: Enhancing Neural Topic Modeling via Maximizing Mutual Information and Group Topic Regularization](https://mlsanity.com/paper/2409.19749/) (2024)
- [Semantic-Driven Topic Modeling Using Transformer-Based Embeddings and Clustering Algorithms](https://mlsanity.com/paper/2410.00134/) (2024)
- [Model Directions, Not Words: Mechanistic Topic Models Using Sparse Autoencoders](https://mlsanity.com/paper/2507.23220/) (2025)
- [A Survey on Retrieval And Structuring Augmented Generation with Large Language Models](https://mlsanity.com/paper/2509.10697/) (2025)
- [Deep de Finetti: Recovering Topic Distributions from Large Language Models](https://mlsanity.com/paper/2312.14226/) (2023)
- [Morality is Non-Binary: Building a Pluralist Moral Sentence Embedding Space using Contrastive Learning](https://mlsanity.com/paper/2401.17228/) (2024)
- [Recent Trends in Linear Text Segmentation: a Survey](https://mlsanity.com/paper/2411.16613/) (2024)
- [Topics as Entity Clusters: Entity-based Topics from Large Language Models and Graph Neural Networks](https://mlsanity.com/paper/2301.02458/) (2023)
- [Lady and the Tramp Nextdoor: Online Manifestations of Economic Inequalities in the Nextdoor Social Network](https://mlsanity.com/paper/2304.05232/) (2023)
- [REAL: A Representative Error-Driven Approach for Active Learning](https://mlsanity.com/paper/2307.00968/) (2023)
- [Comprehensive Evaluation of Large Language Models for Topic Modeling](https://mlsanity.com/paper/2406.00697/) (2024)
- [Refining Dimensions for Improving Clustering-based Cross-lingual Topic Models](https://mlsanity.com/paper/2412.12433/) (2024)
- [ProxAnn: Use-Oriented Evaluations of Topic Models and Document Clustering](https://mlsanity.com/paper/2507.00828/) (2025)
- [Sparse Autoencoders are Topic Models](https://mlsanity.com/paper/2511.16309/) (2025)
- [TroubleLLM: Align to Red Team Expert](https://mlsanity.com/paper/2403.00829/) (2024)
- [Large Language Models for History, Philosophy, and Sociology of Science: Interpretive Uses, Methodological Challenges, and Critical Perspectives](https://mlsanity.com/paper/2506.12242/) (2025)
- [Variational Language Concepts for Interpreting Foundation Language Models](https://mlsanity.com/paper/2410.03964/) (2024)
- [The Medium Is Not the Message: Deconfounding Document Embeddings via Linear Concept Erasure](https://mlsanity.com/paper/2507.01234/) (2025)
- [Moving beyond word lists: towards abstractive topic labels for human-like topics of scientific documents](https://mlsanity.com/paper/2211.05599/) (2022)
- [Graph2topic: an opensource topic modeling framework based on sentence embedding and community detection](https://mlsanity.com/paper/2304.06653/) (2023)
- [Constructing and Interpreting Causal Knowledge Graphs from News](https://mlsanity.com/paper/2305.09359/) (2023)
- [Multilingualism, Transnationality, and K-pop in the Online #StopAsianHate Movement](https://mlsanity.com/paper/2503.02707/) (2025)

## Abstract (arXiv)

Recent work incorporates pre-trained word embeddings such as BERT embeddings into Neural Topic Models (NTMs), generating highly coherent topics. However, with high-quality contextualized document representations, do we really need sophisticated neural models to obtain coherent and interpretable topics? In this paper, we conduct thorough experiments showing that directly clustering high-quality sentence embeddings with an appropriate word selecting method can generate more coherent and diverse topics than NTMs, achieving also higher efficiency and simplicity.