# Contextualized Topic Coherence Metrics

arXiv: https://arxiv.org/abs/2305.14587 · HTML: https://mlsanity.com/paper/2305.14587/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 52 references that are in the map.
- First posted on arXiv 23 May 2023.

## Builds on (most cited first, 16 of 16)

- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [BERTopic: Neural topic modeling with a class-based TF-IDF procedure](https://mlsanity.com/paper/2203.05794/) (2022)
- [Distributed Representations of Sentences and Documents](https://mlsanity.com/paper/1405.4053/) (2014)
- [Topic Modeling in Embedding Spaces](https://mlsanity.com/paper/1907.04907/) (2019)
- [Autoencoding Variational Inference For Topic Models](https://mlsanity.com/paper/1703.01488/) (2017)
- [Top2Vec: Distributed Representations of Topics](https://mlsanity.com/paper/2008.09470/) (2020)
- [Pre-training is a Hot Topic: Contextualized Document Embeddings Improve Topic Coherence](https://mlsanity.com/paper/2004.03974/) (2020)
- [Is Automated Topic Model Evaluation Broken?: The Incoherence of Coherence](https://mlsanity.com/paper/2107.02173/) (2021)
- [Topic Modelling Meets Deep Neural Networks: A Survey](https://mlsanity.com/paper/2103.00498/) (2021)
- [Topic Modeling with Contextualized Word Representation Clusters](https://mlsanity.com/paper/2010.12626/) (2020)
- [ATM:Adversarial-neural Topic Model](https://mlsanity.com/paper/1811.00265/) (2018)
- [Are Neural Topic Models Broken?](https://mlsanity.com/paper/2210.16162/) (2022)
- [Linguistic Dependencies and Statistical Dependence](https://mlsanity.com/paper/2104.08685/) (2021)
- [ANTM: An Aligned Neural Topic Model for Exploring Evolving Topics](https://mlsanity.com/paper/2302.01501/) (2023)
- [Automatic Evaluation of Local Topic Quality](https://mlsanity.com/paper/1905.13126/) (2019)

## Built on by (most cited first, 19 of 19)

- [LLM Reading Tea Leaves: Automatically Evaluating Topic Models with Large Language Models](https://mlsanity.com/paper/2406.09008/) (2024)
- [Neural Topic Modeling with Large Language Models in the Loop](https://mlsanity.com/paper/2411.08534/) (2024)
- [A Computational Method for Measuring "Open Codes" in Qualitative Analysis](https://mlsanity.com/paper/2411.12142/) (2024)
- [PROMPTHEUS: A Human-Centered Pipeline to Streamline SLRs with LLMs](https://mlsanity.com/paper/2410.15978/) (2024)
- [Bridging the Evaluation Gap: Leveraging Large Language Models for Topic Model Evaluation](https://mlsanity.com/paper/2502.07352/) (2025)
- [CAST: Corpus-Aware Self-similarity Enhanced Topic modelling](https://mlsanity.com/paper/2410.15136/) (2024)
- [ProxAnn: Use-Oriented Evaluations of Topic Models and Document Clustering](https://mlsanity.com/paper/2507.00828/) (2025)
- [Sparse Autoencoders are Topic Models](https://mlsanity.com/paper/2511.16309/) (2025)
- [Toward Purpose-oriented Topic Model Evaluation enabled by Large Language Models](https://mlsanity.com/paper/2509.07142/) (2025)
- [Unveiling the Potential of BERTopic for Multilingual Fake News Analysis -- Use Case: Covid-19](https://mlsanity.com/paper/2407.08417/) (2024)
- [Multi-Faceted Question Complexity Estimation Targeting Topic Domain-Specificity](https://mlsanity.com/paper/2408.12850/) (2024)
- [S3 -- Semantic Signal Separation](https://mlsanity.com/paper/2406.09556/) (2024)
- [Contextual Embedding-based Clustering to Identify Topics for Healthcare Service Improvement](https://mlsanity.com/paper/2504.14068/) (2025)
- [Objectifying the Subjective: Cognitive Biases in Topic Interpretations](https://mlsanity.com/paper/2507.19117/) (2025)
- [Improving Topic Modeling by Distilling Soft Labels from Language Models](https://mlsanity.com/paper/2602.17907/) (2026)
- [Disentangling Similarity and Relatedness in Topic Models](https://mlsanity.com/paper/2603.10619/) (2026)
- [From Documents to Segments: A Contextual Reformulation for Topic Assignment](https://mlsanity.com/paper/2605.17714/) (2026)
- [How Do AI Coding Agents Contribute to Software Development? an Empirical Study of Agentic Pull Requests](https://mlsanity.com/paper/2607.21832/) (2026)
- [BERTilda: Explainable Topic Lifecycle Tracking with Split/Merge Detection via Similarity-and-Flow Temporal Graphs](https://mlsanity.com/paper/2608.18101/) (2026)

## Abstract (arXiv)

The recent explosion in work on neural topic modeling has been criticized for optimizing automated topic evaluation metrics at the expense of actual meaningful topic identification. But human annotation remains expensive and time-consuming. We propose LLM-based methods inspired by standard human topic evaluations, in a family of metrics called Contextualized Topic Coherence (CTC). We evaluate both a fully automated version as well as a semi-automated CTC that allows human-centered evaluation of coherence while maintaining the efficiency of automated methods. We evaluate CTC relative to five other metrics on six topic models and find that it outperforms automated topic coherence methods, works well on short documents, and is not susceptible to meaningless but high-scoring topics.