# An Automatic Approach for Document-level Topic Model Evaluation

arXiv: https://arxiv.org/abs/1706.05140 · HTML: https://mlsanity.com/paper/1706.05140/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Jun 2017.

## Built on by (most cited first, 11 of 11)

- [Is Automated Topic Model Evaluation Broken?: The Incoherence of Coherence](https://mlsanity.com/paper/2107.02173/) (2021)
- [LLM Reading Tea Leaves: Automatically Evaluating Topic Models with Large Language Models](https://mlsanity.com/paper/2406.09008/) (2024)
- [Exploring Story Generation with Multi-task Objectives in Variational Autoencoders](https://mlsanity.com/paper/2111.08133/) (2021)
- [Bridging the Evaluation Gap: Leveraging Large Language Models for Topic Model Evaluation](https://mlsanity.com/paper/2502.07352/) (2025)
- [ProxAnn: Use-Oriented Evaluations of Topic Models and Document Clustering](https://mlsanity.com/paper/2507.00828/) (2025)
- [HTMOT : Hierarchical Topic Modelling Over Time](https://mlsanity.com/paper/2112.03104/) (2021)
- [Leveraging Collection-Wide Similarities for Unsupervised Document Structure Extraction](https://mlsanity.com/paper/2402.13906/) (2024)
- [Toward Purpose-oriented Topic Model Evaluation enabled by Large Language Models](https://mlsanity.com/paper/2509.07142/) (2025)
- [Enhance Topics Analysis based on Keywords Properties](https://mlsanity.com/paper/2203.04786/) (2022)
- [Topics in the Haystack: Extracting and Evaluating Topics beyond Coherence](https://mlsanity.com/paper/2303.17324/) (2023)
- [Tell, Don't Show: Leveraging Language Models' Abstractive Retellings to Model Literary Themes](https://mlsanity.com/paper/2505.23166/) (2025)

## Abstract (arXiv)

Topic models jointly learn topics and document-level topic distribution. Extrinsic evaluation of topic models tends to focus exclusively on topic-level evaluation, e.g. by assessing the coherence of topics. We demonstrate that there can be large discrepancies between topic- and document-level model quality, and that basing model evaluation on topic-level analysis can be highly misleading. We propose a method for automatically predicting topic model quality based on analysis of document-level topic allocations, and provide empirical evidence for its robustness.