mlsanityData to 14 Oct 2026

CWTM: Leveraging Contextualized Word Embeddings from BERT for Neural Topic Modeling

arXiv 2305.09329 ↗First posted 16 May 2023cs.CL cs.AIcited by 7

Not on the map. Why?

Abstract (arXiv)

Most existing topic models rely on bag-of-words (BOW) representation, which limits their ability to capture word order information and leads to challenges with out-of-vocabulary (OOV) words in new documents. Contextualized word embeddings, however, show superiority in word sense disambiguation and effectively address the OOV issue. In this work, we introduce a novel neural topic model called the Contextlized Word Topic Model (CWTM), which integrates contextualized word embeddings from BERT. The model is capable of learning the topic vector of a document without BOW information. In addition, it can also derive the topic vectors for individual words within a document based on their contextualized word embeddings. Experiments across various datasets show that CWTM generates more coherent and meaningful topics compared to existing topic models, while also accommodating unseen words in newly encountered documents.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Builds on 26 of its 53 references that are in the map. First posted on arXiv 16 May 2023.

Builds on

26 of 53 references.

References · 26

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  4. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,474
  5. 2022BERTopic: Neural topic modeling with a class-based TF-IDF procedurecited by 801
  6. 2017Wasserstein Auto-Encoderscited by 373
  7. 2019Topic Modeling in Embedding Spacescited by 224
  8. 2017Autoencoding Variational Inference For Topic Modelscited by 169
  9. 2020Pre-training is a Hot Topic: Contextualized Document Embeddings Improve Topic Coherencecited by 121
  10. 2015Neural Variational Inference for Text Processingcited by 111
  11. 2020An Unsupervised Sentence Embedding Method by Mutual Information Maximizationcited by 91
  12. 2020Cross-lingual Contextualized Topic Models with Zero-shot Learningcited by 67
14 more
  1. 2019Topic Modeling with Wasserstein Autoencoderscited by 53
  2. 2020Topic Modeling with Contextualized Word Representation Clusterscited by 41
  3. 2022Topic Discovery via Latent Space Clustering of Pretrained Language Model Representationscited by 34
  4. 2022Improved Universal Sentence Embeddings with Prompt-based Contrastive Learning and Energy-based Learningcited by 30
  5. 2018Improving Topic Models with Latent Feature Word Representationscited by 28
  6. 2018Coherence-Aware Neural Topic Modelingcited by 26
  7. 2020Improving Neural Topic Models using Knowledge Distillationcited by 25
  8. 2020Neural Topic Modeling with Continual Lifelong Learningcited by 24
  9. 2018Document Informed Neural Autoregressive Topic Models with Distributional Priorcited by 11
  10. 2021Fine-tuning Encoders for Improved Monolingual and Zero-shot Polylingual Neural Topic Modelingcited by 8
  11. 2017Jointly Learning Word Embeddings and Latent Topicscited by 6
  12. 2017MetaLDA: a Topic Model that Efficiently Incorporates Meta informationcited by 6
  13. 2020A Neural Generative Model for Joint Learning Topics and Topic-Specific Word Embeddingscited by 3
  14. 2017Mixed Membership Word Embeddings for Computational Social Sciencecited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 5.

7 papers.

  1. 2025CEMTM: Contextual Embedding-based Multimodal Topic Modelingcited by 4
  2. 2025Topic-Guided Reinforcement Learning with LLMs for Enhancing Multi-Document Summarizationcited by 0
  3. 2026Improving Topic Modeling by Distilling Soft Labels from Language Modelscited by 0
  4. 2026Disentangling Similarity and Relatedness in Topic Modelscited by 0
  5. 2026Human-Centric Topic Modeling with Goal-Prompted Contrastive Learning and Optimal Transportcited by 0
  6. 2026A comparative study of transformer-based embeddings for topic coherencecited by 0
  7. 2026MCompassRAG: Topic Metadata as a Semantic Compass for Paragraph-Level Retrievalcited by 0