# Revisiting Automated Topic Model Evaluation with Large Language Models

arXiv: https://arxiv.org/abs/2305.12152 · HTML: https://mlsanity.com/paper/2305.12152/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 39 references that are in the map.
- First posted on arXiv 20 May 2023.

## Builds on (most cited first, 11 of 11)

- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [ChatGPT Outperforms Crowd-Workers for Text-Annotation Tasks](https://mlsanity.com/paper/2303.15056/) (2023)
- [Large Language Models Are State-of-the-Art Evaluators of Translation Quality](https://mlsanity.com/paper/2302.14520/) (2023)
- [Persistent Anti-Muslim Bias in Large Language Models](https://mlsanity.com/paper/2101.05783/) (2021)
- [GPTScore: Evaluate as You Desire](https://mlsanity.com/paper/2302.04166/) (2023)
- [Topic Modeling in Embedding Spaces](https://mlsanity.com/paper/1907.04907/) (2019)
- [Is ChatGPT better than Human Annotators? Potential and Limitations of ChatGPT in Explaining Implicit Hate Speech](https://mlsanity.com/paper/2302.07736/) (2023)
- [Perspectives on Large Language Models for Relevance Judgment](https://mlsanity.com/paper/2304.09161/) (2023)
- [Is Automated Topic Model Evaluation Broken?: The Incoherence of Coherence](https://mlsanity.com/paper/2107.02173/) (2021)
- [Goal Driven Discovery of Distributional Differences via Language Descriptions](https://mlsanity.com/paper/2302.14233/) (2023)

## Built on by (most cited first, 3 of 3)

- [Unveiling the Potential of BERTopic for Multilingual Fake News Analysis -- Use Case: Covid-19](https://mlsanity.com/paper/2407.08417/) (2024)
- [Enhancing Topic Interpretability for Neural Topic Modeling through Topic-wise Contrastive Learning](https://mlsanity.com/paper/2412.17338/) (2024)
- [LLM-Assisted Topic Reduction for BERTopic on Social Media Data](https://mlsanity.com/paper/2509.19365/) (2025)

## Abstract (arXiv)

Topic models are used to make sense of large text collections. However, automatically evaluating topic model output and determining the optimal number of topics both have been longstanding challenges, with no effective automated solutions to date. This paper proposes using large language models to evaluate such output. We find that large language models appropriately assess the resulting topics, correlating more strongly with human judgments than existing automated metrics. We then investigate whether we can use large language models to automatically determine the optimal number of topics. We automatically assign labels to documents and choosing configurations with the most pure labels returns reasonable values for the optimal number of topics.