# CWTM: Leveraging Contextualized Word Embeddings from BERT for Neural Topic Modeling

arXiv: https://arxiv.org/abs/2305.09329 · HTML: https://mlsanity.com/paper/2305.09329/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 53 references that are in the map.
- First posted on arXiv 16 May 2023.

## Builds on (most cited first, 26 of 26)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [BERTopic: Neural topic modeling with a class-based TF-IDF procedure](https://mlsanity.com/paper/2203.05794/) (2022)
- [Wasserstein Auto-Encoders](https://mlsanity.com/paper/1711.01558/) (2017)
- [Topic Modeling in Embedding Spaces](https://mlsanity.com/paper/1907.04907/) (2019)
- [Autoencoding Variational Inference For Topic Models](https://mlsanity.com/paper/1703.01488/) (2017)
- [Pre-training is a Hot Topic: Contextualized Document Embeddings Improve Topic Coherence](https://mlsanity.com/paper/2004.03974/) (2020)
- [Neural Variational Inference for Text Processing](https://mlsanity.com/paper/1511.06038/) (2015)
- [An Unsupervised Sentence Embedding Method by Mutual Information Maximization](https://mlsanity.com/paper/2009.12061/) (2020)
- [Cross-lingual Contextualized Topic Models with Zero-shot Learning](https://mlsanity.com/paper/2004.07737/) (2020)
- [Topic Modeling with Wasserstein Autoencoders](https://mlsanity.com/paper/1907.12374/) (2019)
- [Topic Modeling with Contextualized Word Representation Clusters](https://mlsanity.com/paper/2010.12626/) (2020)
- [Topic Discovery via Latent Space Clustering of Pretrained Language Model Representations](https://mlsanity.com/paper/2202.04582/) (2022)
- [Improved Universal Sentence Embeddings with Prompt-based Contrastive Learning and Energy-based Learning](https://mlsanity.com/paper/2203.06875/) (2022)
- [Improving Topic Models with Latent Feature Word Representations](https://mlsanity.com/paper/1810.06306/) (2018)
- [Coherence-Aware Neural Topic Modeling](https://mlsanity.com/paper/1809.02687/) (2018)
- [Improving Neural Topic Models using Knowledge Distillation](https://mlsanity.com/paper/2010.02377/) (2020)
- [Neural Topic Modeling with Continual Lifelong Learning](https://mlsanity.com/paper/2006.10909/) (2020)
- [Document Informed Neural Autoregressive Topic Models with Distributional Prior](https://mlsanity.com/paper/1809.06709/) (2018)
- [Fine-tuning Encoders for Improved Monolingual and Zero-shot Polylingual Neural Topic Modeling](https://mlsanity.com/paper/2104.05064/) (2021)
- [Jointly Learning Word Embeddings and Latent Topics](https://mlsanity.com/paper/1706.07276/) (2017)
- [MetaLDA: a Topic Model that Efficiently Incorporates Meta information](https://mlsanity.com/paper/1709.06365/) (2017)
- [A Neural Generative Model for Joint Learning Topics and Topic-Specific Word Embeddings](https://mlsanity.com/paper/2008.04702/) (2020)
- [Mixed Membership Word Embeddings for Computational Social Science](https://mlsanity.com/paper/1705.07368/) (2017)

## Built on by (most cited first, 7 of 7)

- [CEMTM: Contextual Embedding-based Multimodal Topic Modeling](https://mlsanity.com/paper/2509.11465/) (2025)
- [Topic-Guided Reinforcement Learning with LLMs for Enhancing Multi-Document Summarization](https://mlsanity.com/paper/2509.09852/) (2025)
- [Improving Topic Modeling by Distilling Soft Labels from Language Models](https://mlsanity.com/paper/2602.17907/) (2026)
- [Disentangling Similarity and Relatedness in Topic Models](https://mlsanity.com/paper/2603.10619/) (2026)
- [Human-Centric Topic Modeling with Goal-Prompted Contrastive Learning and Optimal Transport](https://mlsanity.com/paper/2604.12663/) (2026)
- [A comparative study of transformer-based embeddings for topic coherence](https://mlsanity.com/paper/2605.28832/) (2026)
- [MCompassRAG: Topic Metadata as a Semantic Compass for Paragraph-Level Retrieval](https://mlsanity.com/paper/2606.18508/) (2026)

## Abstract (arXiv)

Most existing topic models rely on bag-of-words (BOW) representation, which limits their ability to capture word order information and leads to challenges with out-of-vocabulary (OOV) words in new documents. Contextualized word embeddings, however, show superiority in word sense disambiguation and effectively address the OOV issue. In this work, we introduce a novel neural topic model called the Contextlized Word Topic Model (CWTM), which integrates contextualized word embeddings from BERT. The model is capable of learning the topic vector of a document without BOW information. In addition, it can also derive the topic vectors for individual words within a document based on their contextualized word embeddings. Experiments across various datasets show that CWTM generates more coherent and meaningful topics compared to existing topic models, while also accommodating unseen words in newly encountered documents.