# Topic Modeling with Contextualized Word Representation Clusters

arXiv: https://arxiv.org/abs/2010.12626 · HTML: https://mlsanity.com/paper/2010.12626/

## Key facts

- Cited by 41 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Oct 2020.

## Built on by (most cited first, 30 of 41)

- [BERTopic: Neural topic modeling with a class-based TF-IDF procedure](https://mlsanity.com/paper/2203.05794/) (2022)
- [How Do Transformers Learn Topic Structure: Towards a Mechanistic Understanding](https://mlsanity.com/paper/2303.04245/) (2023)
- [Is Automated Topic Model Evaluation Broken?: The Incoherence of Coherence](https://mlsanity.com/paper/2107.02173/) (2021)
- [Topic Modelling Meets Deep Neural Networks: A Survey](https://mlsanity.com/paper/2103.00498/) (2021)
- [Is Neural Topic Modelling Better than Clustering? An Empirical Study on Clustering with Contextual Embeddings for Topics](https://mlsanity.com/paper/2204.09874/) (2022)
- [Topic Discovery via Latent Space Clustering of Pretrained Language Model Representations](https://mlsanity.com/paper/2202.04582/) (2022)
- [On Behalf of the Stakeholders: Trends in NLP Model Interpretability in the Era of LLMs](https://mlsanity.com/paper/2407.19200/) (2024)
- [Contextualized Topic Coherence Metrics](https://mlsanity.com/paper/2305.14587/) (2023)
- [Uncovering hidden geometry in Transformers via disentangling position and context](https://mlsanity.com/paper/2310.04861/) (2023)
- [A Survey on Neural Topic Models: Methods, Applications, and Challenges](https://mlsanity.com/paper/2401.15351/) (2024)
- [Effective Seed-Guided Topic Discovery by Integrating Multiple Types of Contexts](https://mlsanity.com/paper/2212.06002/) (2022)
- [Scaling up Discovery of Latent Concepts in Deep NLP Models](https://mlsanity.com/paper/2308.10263/) (2023)
- [Seed-Guided Topic Discovery with Out-of-Vocabulary Seeds](https://mlsanity.com/paper/2205.01845/) (2022)
- [CWTM: Leveraging Contextualized Word Embeddings from BERT for Neural Topic Modeling](https://mlsanity.com/paper/2305.09329/) (2023)
- [Artificial intelligence for topic modelling in Hindu philosophy: mapping themes between the Upanishads and the Bhagavad Gita](https://mlsanity.com/paper/2205.11020/) (2022)
- [ANTM: An Aligned Neural Topic Model for Exploring Evolving Topics](https://mlsanity.com/paper/2302.01501/) (2023)
- [A Meta-Learning Perspective on Transformers for Causal Language Modeling](https://mlsanity.com/paper/2310.05884/) (2023)
- [Transformer-based Causal Language Models Perform Clustering](https://mlsanity.com/paper/2402.12151/) (2024)
- [Large-scale cloze evaluation reveals that token prediction tasks are neither lexically nor semantically aligned](https://mlsanity.com/paper/2410.12057/) (2024)
- [Searching for Structure in Unfalsifiable Claims](https://mlsanity.com/paper/2209.00495/) (2022)
- [Deep learning for COVID-19 topic modelling via Twitter: Alpha, Delta and Omicron](https://mlsanity.com/paper/2303.00135/) (2023)
- [Topics as Entity Clusters: Entity-based Topics from Large Language Models and Graph Neural Networks](https://mlsanity.com/paper/2301.02458/) (2023)
- [Sparse Autoencoders are Topic Models](https://mlsanity.com/paper/2511.16309/) (2025)
- [Deriving Word Vectors from Contextualized Language Models using Topic-Aware Mention Selection](https://mlsanity.com/paper/2106.07947/) (2021)
- [Forecasting COVID-19 Caseloads Using Unsupervised Embedding Clusters of Social Media Posts](https://mlsanity.com/paper/2205.10408/) (2022)
- [Mapping of Subjective Accounts into Interpreted Clusters (MOSAIC): Topic Modelling and LLM applied to Stroboscopic Phenomenology](https://mlsanity.com/paper/2502.18318/) (2025)
- [Survey of Generative Methods for Social Media Analysis](https://mlsanity.com/paper/2112.07041/) (2021)
- [Topics in Contextualised Attention Embeddings](https://mlsanity.com/paper/2301.04339/) (2023)
- [FATRER: Full-Attention Topic Regularizer for Accurate and Robust Conversational Emotion Recognition](https://mlsanity.com/paper/2307.12221/) (2023)
- [A Contextual Topic Modeling and Content Analysis of Iranian laws and Regulations](https://mlsanity.com/paper/2309.13051/) (2023)

## Abstract (arXiv)

Clustering token-level contextualized word representations produces output that shares many similarities with topic models for English text collections. Unlike clusterings of vocabulary-level word embeddings, the resulting models more naturally capture polysemy and can be used as a way of organizing documents. We evaluate token clusterings trained from several different output layers of popular contextualized language models. We find that BERT and GPT-2 produce high quality clusterings, but RoBERTa does not. These cluster models are simple, reliable, and can perform as well as, if not better than, LDA topic models, maintaining high topic quality even when the number of topics is large relative to the size of the local collection.