Toward Purpose-oriented Topic Model Evaluation enabled by Large Language Models
Not on the map. Why?
Abstract (arXiv)
This study presents a framework for automated evaluation of dynamically evolving topic models using Large Language Models (LLMs). Topic modeling is essential for organizing and retrieving scholarly content in digital library systems, helping users navigate complex and evolving knowledge domains. However, widely used automated metrics, such as coherence and diversity, often capture only narrow statistical patterns and fail to explain semantic failures in practice. We introduce a purpose-oriented evaluation framework that employs nine LLM-based metrics spanning four key dimensions of topic quality: lexical validity, intra-topic semantic soundness, inter-topic structural soundness, and document-topic alignment soundness. The framework is validated through adversarial and sampling-based protocols, and is applied across datasets spanning news articles, scholarly publications, and social media posts, as well as multiple topic modeling methods and open-source LLMs. Our analysis shows that LLM-based metrics provide interpretable, robust, and task-relevant assessments, uncovering critical weaknesses in topic models such as redundancy and semantic drift, which are often missed by traditional metrics. These results support the development of scalable, fine-grained evaluation tools for maintaining topic relevance in dynamic datasets. All code and data supporting this work are accessible at https://github.com/zhiyintan/topic-model-LLMjudgment.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 23 of its 71 references that are in the map. First posted on arXiv 8 Sep 2025.
Builds on
23 of 71 references.
References · 23
- 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
- 2022BERTopic: Neural topic modeling with a class-based TF-IDF procedurecited by 801
- 2019Topic Modeling in Embedding Spacescited by 224
- 2017Autoencoding Variational Inference For Topic Modelscited by 169
- 2020Pre-training is a Hot Topic: Contextualized Document Embeddings Improve Topic Coherencecited by 121
- 2017Discovering Discrete Latent Topics with Neural Variational Inferencecited by 79
- 2020Tired of Topic Models? Clusters of Pretrained Word Embeddings Make for Fast and Good Topics too!cited by 73
- 2020Cross-lingual Contextualized Topic Models with Zero-shot Learningcited by 67
- 2019Topic Modeling with Wasserstein Autoencoderscited by 53
- 2020Neural Topic Model via Optimal Transportcited by 50
- 2017Neural Models for Documents with Metadatacited by 46
- 2023Effective Neural Topic Modeling with Embedding Clustering Regularizationcited by 46
11 more
- 2024Segment Any Text: A Universal Approach for Robust, Efficient and Adaptable Sentence Segmentationcited by 41
- 2018WHAI: Weibull Hybrid Autoencoding Inference for Deep Topic Modelingcited by 32
- 2020Neural Topic Modeling with Bidirectional Adversarial Trainingcited by 29
- 2018Improving Topic Models with Latent Feature Word Representationscited by 28
- 2018Coherence-Aware Neural Topic Modelingcited by 26
- 2018ATM:Adversarial-neural Topic Modelcited by 23
- 2023Contextualized Topic Coherence Metricscited by 19
- 2017An Automatic Approach for Document-level Topic Model Evaluationcited by 11
- 2024LLM Reading Tea Leaves: Automatically Evaluating Topic Models with Large Language Modelscited by 11
- 2020A Topic Coverage Approach to Evaluation of Topic Modelscited by 7
- 2025Bridging the Evaluation Gap: Leveraging Large Language Models for Topic Model Evaluationcited by 4
Built on by
Citing papers per year: 2026: 2.
2 papers.