mlsanityData to 7 Oct 2026

Topic Modeling with Contextualized Word Representation Clusters

arXiv 2010.12626 ↗First posted 23 Oct 2020cs.CLcited by 41

Not on the map. Why?

Abstract (arXiv)

Clustering token-level contextualized word representations produces output that shares many similarities with topic models for English text collections. Unlike clusterings of vocabulary-level word embeddings, the resulting models more naturally capture polysemy and can be used as a way of organizing documents. We evaluate token clusterings trained from several different output layers of popular contextualized language models. We find that BERT and GPT-2 produce high quality clusterings, but RoBERTa does not. These cluster models are simple, reliable, and can perform as well as, if not better than, LDA topic models, maintaining high topic quality even when the number of topics is large relative to the size of the local collection.

Key facts. Cited by 41 papers in the map (data through 7 Oct 2026). First posted on arXiv 23 Oct 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 4; 2022: 9; 2023: 15; 2024: 8; 2025: 4; 2026: 1.

41 papers.

Sort:

By area:

  1. 2022BERTopic: Neural topic modeling with a class-based TF-IDF procedurecited by 779
  2. 2023How Do Transformers Learn Topic Structure: Towards a Mechanistic Understandingcited by 85
  3. 2021Is Automated Topic Model Evaluation Broken?: The Incoherence of Coherencecited by 74
  4. 2021Topic Modelling Meets Deep Neural Networks: A Surveycited by 51
  5. 2022Is Neural Topic Modelling Better than Clustering? An Empirical Study on Clustering with Contextual Embeddings for Topicscited by 35
  6. 2022Topic Discovery via Latent Space Clustering of Pretrained Language Model Representationscited by 34
  7. 2024On Behalf of the Stakeholders: Trends in NLP Model Interpretability in the Era of LLMscited by 24
  8. 2023Contextualized Topic Coherence Metricscited by 19
  9. 2024A Survey on Neural Topic Models: Methods, Applications, and Challengescited by 18
  10. 2023Uncovering hidden geometry in Transformers via disentangling position and contextcited by 15
  11. 2022Effective Seed-Guided Topic Discovery by Integrating Multiple Types of Contextscited by 12
  12. 2023Scaling up Discovery of Latent Concepts in Deep NLP Modelscited by 9
29 more
  1. 2022Seed-Guided Topic Discovery with Out-of-Vocabulary Seedscited by 7
  2. 2023CWTM: Leveraging Contextualized Word Embeddings from BERT for Neural Topic Modelingcited by 7
  3. 2022Artificial intelligence for topic modelling in Hindu philosophy: mapping themes between the Upanishads and the Bhagavad Gitacited by 6
  4. 2023ANTM: An Aligned Neural Topic Model for Exploring Evolving Topicscited by 6
  5. 2023A Meta-Learning Perspective on Transformers for Causal Language Modelingcited by 5
  6. 2024Transformer-based Causal Language Models Perform Clusteringcited by 5
  7. 2022Searching for Structure in Unfalsifiable Claimscited by 4
  8. 2023Deep learning for COVID-19 topic modelling via Twitter: Alpha, Delta and Omicroncited by 4
  9. 2024Large-scale cloze evaluation reveals that token prediction tasks are neither lexically nor semantically alignedcited by 4
  10. 2023Topics as Entity Clusters: Entity-based Topics from Large Language Models and Graph Neural Networkscited by 3
  11. 2025Sparse Autoencoders are Topic Modelscited by 3
  12. 2021Deriving Word Vectors from Contextualized Language Models using Topic-Aware Mention Selectioncited by 2
  13. 2022Forecasting COVID-19 Caseloads Using Unsupervised Embedding Clusters of Social Media Postscited by 2
  14. 2025Mapping of Subjective Accounts into Interpreted Clusters (MOSAIC): Topic Modelling and LLM applied to Stroboscopic Phenomenologycited by 2
  15. 2021Survey of Generative Methods for Social Media Analysiscited by 1
  16. 2023Topics in Contextualised Attention Embeddingscited by 1
  17. 2023FATRER: Full-Attention Topic Regularizer for Accurate and Robust Conversational Emotion Recognitioncited by 1
  18. 2023A Contextual Topic Modeling and Content Analysis of Iranian laws and Regulationscited by 1
  19. 2023TopicGPT: A Prompt-based Topic Modeling Frameworkcited by 1
  20. 2023Dynamic Topic Language Model on Heterogeneous Children's Mental Health Clinical Notescited by 1
  21. 2024Interactive Topic Models with Optimal Transportcited by 1
  22. 2024Capturing research literature attitude towards Sustainable Development Goals: an LLM-based topic modeling approachcited by 1
  23. 2022Automatic Scene-based Topic Channel Construction System for E-Commercecited by 0
  24. 2023Latent Feature-based Data Splits to Improve Generalisation Evaluation: A Hate Speech Detection Case Studycited by 0
  25. 2024Deciphering public attention to geoengineering and climate issues using machine learning and dynamic analysiscited by 0
  26. 2024An Iterative Approach to Topic Modellingcited by 0
  27. 2025Multilingualism, Transnationality, and K-pop in the Online #StopAsianHate Movementcited by 0
  28. 2025Contextual Embedding-based Clustering to Identify Topics for Healthcare Service Improvementcited by 0
  29. 2026Disentangling Similarity and Relatedness in Topic Modelscited by 0