# Topic Discovery via Latent Space Clustering of Pretrained Language Model Representations

arXiv: https://arxiv.org/abs/2202.04582 · HTML: https://mlsanity.com/paper/2202.04582/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- Builds on 37 of its 69 references that are in the map.
- First posted on arXiv 9 Feb 2022.

## Builds on (most cited first, 30 of 37)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models](https://mlsanity.com/paper/2009.11462/) (2020)
- [Unsupervised Deep Embedding for Clustering Analysis](https://mlsanity.com/paper/1511.06335/) (2015)
- [Pre-trained Models for Natural Language Processing: A Survey](https://mlsanity.com/paper/2003.08271/) (2020)
- [Topic Modeling in Embedding Spaces](https://mlsanity.com/paper/1907.04907/) (2019)
- [Autoencoding Variational Inference For Topic Models](https://mlsanity.com/paper/1703.01488/) (2017)
- [COCO-LM: Correcting and Contrasting Text Sequences for Language Model Pretraining](https://mlsanity.com/paper/2102.08473/) (2021)
- [Style Transfer Through Back-Translation](https://mlsanity.com/paper/1804.09000/) (2018)
- [Pre-training is a Hot Topic: Contextualized Document Embeddings Improve Topic Coherence](https://mlsanity.com/paper/2004.03974/) (2020)
- [Neural Variational Inference for Text Processing](https://mlsanity.com/paper/1511.06038/) (2015)
- [On the Sentence Embeddings from Pre-trained Language Models](https://mlsanity.com/paper/2011.05864/) (2020)
- [Tired of Topic Models? Clusters of Pretrained Word Embeddings Make for Fast and Good Topics too!](https://mlsanity.com/paper/2004.14914/) (2020)
- [Weakly-Supervised Neural Text Classification](https://mlsanity.com/paper/1809.01478/) (2018)
- [PowerTransformer: Unsupervised Controllable Revision for Biased Language Correction](https://mlsanity.com/paper/2010.13816/) (2020)
- [Spherical Text Embedding](https://mlsanity.com/paper/1911.01196/) (2019)
- [Weakly-Supervised Hierarchical Text Classification](https://mlsanity.com/paper/1812.11270/) (2018)
- [Topic Modeling with Contextualized Word Representation Clusters](https://mlsanity.com/paper/2010.12626/) (2020)
- [Text Classification Using Label Names Only: A Language Model Self-Training Approach](https://mlsanity.com/paper/2010.07245/) (2020)
- [Discriminative Topic Mining via Category-Name Guided Text Embedding](https://mlsanity.com/paper/1908.07162/) (2019)
- [Neural Topic Modeling with Bidirectional Adversarial Training](https://mlsanity.com/paper/2004.12331/) (2020)
- [Improving Topic Models with Latent Feature Word Representations](https://mlsanity.com/paper/1810.06306/) (2018)
- [Improving Neural Topic Models using Knowledge Distillation](https://mlsanity.com/paper/2010.02377/) (2020)
- [Anchored Correlation Explanation: Topic Modeling with Minimal Domain Knowledge](https://mlsanity.com/paper/1611.10277/) (2016)

## Built on by (most cited first, 30 of 34)

- [How Do Transformers Learn Topic Structure: Towards a Mechanistic Understanding](https://mlsanity.com/paper/2303.04245/) (2023)
- [Prompting Large Language Models for Topic Modeling](https://mlsanity.com/paper/2312.09693/) (2023)
- [Heterformer: Transformer-based Deep Node Representation Learning on Heterogeneous Text-Rich Networks](https://mlsanity.com/paper/2205.10282/) (2022)
- [SEAL : Interactive Tool for Systematic Error Analysis and Labeling](https://mlsanity.com/paper/2210.05839/) (2022)
- [Effective Seed-Guided Topic Discovery by Integrating Multiple Types of Contexts](https://mlsanity.com/paper/2212.06002/) (2022)
- [Comparison of Topic Modelling Approaches in the Banking Context](https://mlsanity.com/paper/2402.03176/) (2024)
- [What Should Embeddings Embed? Autoregressive Models Represent Latent Generating Distributions](https://mlsanity.com/paper/2406.03707/) (2024)
- [Context-Aware Clustering using Large Language Models](https://mlsanity.com/paper/2405.00988/) (2024)
- [Voices in a Crowd: Searching for Clusters of Unique Perspectives](https://mlsanity.com/paper/2407.14259/) (2024)
- [Emergence of Hierarchical Emotion Organization in Large Language Models](https://mlsanity.com/paper/2507.10599/) (2025)
- [A Survey on Retrieval And Structuring Augmented Generation with Large Language Models](https://mlsanity.com/paper/2509.10697/) (2025)
- [CWTM: Leveraging Contextualized Word Embeddings from BERT for Neural Topic Modeling](https://mlsanity.com/paper/2305.09329/) (2023)
- [Specious Sites: Tracking the Spread and Sway of Spurious News Stories at Scale](https://mlsanity.com/paper/2308.02068/) (2023)
- [Deep de Finetti: Recovering Topic Distributions from Large Language Models](https://mlsanity.com/paper/2312.14226/) (2023)
- [Topic-VQ-VAE: Leveraging Latent Codebooks for Flexible Topic-Guided Document Generation](https://mlsanity.com/paper/2312.11532/) (2023)
- [MEGClass: Extremely Weakly Supervised Text Classification via Mutually-Enhancing Text Granularities](https://mlsanity.com/paper/2304.01969/) (2023)
- [Self-supervised Topic Taxonomy Discovery in the Box Embedding Space](https://mlsanity.com/paper/2408.15050/) (2024)
- [CAST: Corpus-Aware Self-similarity Enhanced Topic modelling](https://mlsanity.com/paper/2410.15136/) (2024)
- [Topic Modeling with Fine-tuning LLMs and Bag of Sentences](https://mlsanity.com/paper/2408.03099/) (2024)
- [HierPromptLM: A Pure PLM-based Framework for Representation Learning on Heterogeneous Text-rich Networks](https://mlsanity.com/paper/2501.12857/) (2025)
- [Efficient and Flexible Topic Modeling using Pretrained Embeddings and Bag of Sentences](https://mlsanity.com/paper/2302.03106/) (2023)
- [Exploring the evolution of research topics during the COVID-19 pandemic](https://mlsanity.com/paper/2310.03928/) (2023)
- [SubgroupTE: Advancing Treatment Effect Estimation with Subgroup Identification](https://mlsanity.com/paper/2401.12369/) (2024)
- [Variational Language Concepts for Interpreting Foundation Language Models](https://mlsanity.com/paper/2410.03964/) (2024)
- [Capturing research literature attitude towards Sustainable Development Goals: an LLM-based topic modeling approach](https://mlsanity.com/paper/2411.02943/) (2024)
- [Tracking the Takes and Trajectories of English-Language News Narratives across Trustworthy and Worrisome Websites](https://mlsanity.com/paper/2501.09102/) (2025)
- [Poisson-Process Topic Model for Integrating Knowledge from Pre-trained Language Models](https://mlsanity.com/paper/2503.17809/) (2025)
- [RepCali: High Efficient Fine-tuning Via Representation Calibration in Latent Space for Pre-trained Language Models](https://mlsanity.com/paper/2505.08463/) (2025)
- [Cone: Unsupervised Contrastive Opinion Extraction](https://mlsanity.com/paper/2305.04599/) (2023)
- [Topic-Guided Self-Introduction Generation for Social Media Users](https://mlsanity.com/paper/2305.15138/) (2023)

## Abstract (arXiv)

Topic models have been the prominent tools for automatic topic discovery from text corpora. Despite their effectiveness, topic models suffer from several limitations including the inability of modeling word ordering information in documents, the difficulty of incorporating external linguistic knowledge, and the lack of both accurate and efficient inference methods for approximating the intractable posterior. Recently, pretrained language models (PLMs) have brought astonishing performance improvements to a wide variety of tasks due to their superior representations of text. Interestingly, there have not been standard approaches to deploy PLMs for topic discovery as better alternatives to topic models. In this paper, we begin by analyzing the challenges of using PLM representations for topic discovery, and then propose a joint latent space learning and clustering framework built upon PLM embeddings. In the latent space, topic-word and document-topic distributions are jointly modeled so that the discovered topics can be interpreted by coherent and distinctive terms and meanwhile serve as meaningful summaries of the documents. Our model effectively leverages the strong representation power and superb linguistic features brought by PLMs for topic discovery, and is conceptually simpler than topic models. On two benchmark datasets in different domains, our model generates significantly more coherent and diverse topics than strong topic models, and offers better topic-wise document representations, based on both automatic and human evaluations.