# CAST: Corpus-Aware Self-similarity Enhanced Topic modelling

arXiv: https://arxiv.org/abs/2410.15136 · HTML: https://mlsanity.com/paper/2410.15136/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 39 references that are in the map.
- First posted on arXiv 19 Oct 2024.

## Builds on (most cited first, 19 of 19)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction](https://mlsanity.com/paper/1802.03426/) (2018)
- [SimCSE: Simple Contrastive Learning of Sentence Embeddings](https://mlsanity.com/paper/2104.08821/) (2021)
- [BERTopic: Neural topic modeling with a class-based TF-IDF procedure](https://mlsanity.com/paper/2203.05794/) (2022)
- [How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings](https://mlsanity.com/paper/1909.00512/) (2019)
- [Topic Modeling in Embedding Spaces](https://mlsanity.com/paper/1907.04907/) (2019)
- [Top2Vec: Distributed Representations of Topics](https://mlsanity.com/paper/2008.09470/) (2020)
- [Is Automated Topic Model Evaluation Broken?: The Incoherence of Coherence](https://mlsanity.com/paper/2107.02173/) (2021)
- [Tired of Topic Models? Clusters of Pretrained Word Embeddings Make for Fast and Good Topics too!](https://mlsanity.com/paper/2004.14914/) (2020)
- [Large Language Models Offer an Alternative to the Traditional Approach of Topic Modelling](https://mlsanity.com/paper/2403.16248/) (2024)
- [Topic Discovery via Latent Space Clustering of Pretrained Language Model Representations](https://mlsanity.com/paper/2202.04582/) (2022)
- [Prompting Large Language Models for Topic Modeling](https://mlsanity.com/paper/2312.09693/) (2023)
- [Contextualized Topic Coherence Metrics](https://mlsanity.com/paper/2305.14587/) (2023)
- [On Isotropy, Contextualization and Learning Dynamics of Contrastive-based Sentence Representation Learning](https://mlsanity.com/paper/2212.09170/) (2022)
- [Length is a Curse and a Blessing for Document-level Semantics](https://mlsanity.com/paper/2310.16193/) (2023)

## Built on by (most cited first, 3 of 3)

- [Human-Centric Topic Modeling with Goal-Prompted Contrastive Learning and Optimal Transport](https://mlsanity.com/paper/2604.12663/) (2026)
- [A Manifold-Aware Topic Modeling Approach via Rank-Based Prototypes](https://mlsanity.com/paper/2609.29630/) (2026)
- [Look What You Made Us Cluster: Hate Narrative Extraction from Reddit Discourse](https://mlsanity.com/paper/2609.37408/) (2026)

## Abstract (arXiv)

Topic modelling is a pivotal unsupervised machine learning technique for extracting valuable insights from large document collections. Existing neural topic modelling methods often encode contextual information of documents, while ignoring contextual details of candidate centroid words, leading to the inaccurate selection of topic words due to the contextualization gap. In parallel, it is found that functional words are frequently selected over topical words. To address these limitations, we introduce CAST: Corpus-Aware Self-similarity Enhanced Topic modelling, a novel topic modelling method that builds upon candidate centroid word embeddings contextualized on the dataset, and a novel self-similarity-based method to filter out less meaningful tokens. Inspired by findings in contrastive learning that self-similarities of functional token embeddings in different contexts are much lower than topical tokens, we find self-similarity to be an effective metric to prevent functional words from acting as candidate topic words. Our approach significantly enhances the coherence and diversity of generated topics, as well as the topic model's ability to handle noisy data. Experiments on news benchmark datasets and one Twitter dataset demonstrate the method's superiority in generating coherent, diverse topics, and handling noisy data, outperforming strong baselines.