mlsanityData to 7 Oct 2026

LLM Reading Tea Leaves: Automatically Evaluating Topic Models with Large Language Models

arXiv 2406.09008 ↗First posted 13 Jun 2024cs.CLcited by 11

Not on the map. Why?

Abstract (arXiv)

Topic modeling has been a widely used tool for unsupervised text analysis. However, comprehensive evaluations of a topic model remain challenging. Existing evaluation methods are either less comparable across different models (e.g., perplexity) or focus on only one specific aspect of a model (e.g., topic quality or document representation quality) at a time, which is insufficient to reflect the overall model performance. In this paper, we propose WALM (Word Agreement with Language Model), a new evaluation method for topic modeling that considers the semantic quality of document representations and topics in a joint manner, leveraging the power of Large Language Models (LLMs). With extensive experiments involving different types of topic models, WALM is shown to align with human judgment and can serve as a complementary evaluation method to the existing ones, bringing a new perspective to topic modeling. Our software package is available at https://github.com/Xiaohao-Yang/Topic_Model_Evaluation.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Builds on 38 of its 75 references that are in the map. First posted on arXiv 13 Jun 2024.

Builds on

38 of 75 references.

References · 38

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  5. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,346
  6. 2023Mistral 7Bcited by 3,065
  7. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
  8. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  9. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,184
  10. 2022BERTopic: Neural topic modeling with a class-based TF-IDF procedurecited by 779
  11. 2024Yi: Open Foundation Models by 01.AIcited by 706
  12. 2023Benchmarking Large Language Models for News Summarizationcited by 444
26 more
  1. 2019Topic Modeling in Embedding Spacescited by 224
  2. 2021OTA: Optimal Transport Assignment for Object Detectioncited by 177
  3. 2017Autoencoding Variational Inference For Topic Modelscited by 168
  4. 2019Don't Blame the ELBO! A Linear VAE Perspective on Posterior Collapsecited by 120
  5. 2017Discovering Discrete Latent Topics with Neural Variational Inferencecited by 79
  6. 2020Tired of Topic Models? Clusters of Pretrained Word Embeddings Make for Fast and Good Topics too!cited by 73
  7. 2021Topic Modelling Meets Deep Neural Networks: A Surveycited by 51
  8. 2020Neural Topic Model via Optimal Transportcited by 50
  9. 2021Contrastive Learning for Neural Topic Modelcited by 48
  10. 2017Neural Models for Documents with Metadatacited by 46
  11. 2022A Unified Wasserstein Distributional Robustness Framework for Adversarial Trainingcited by 36
  12. 2022Representing Mixtures of Word Embeddings with Mixtures of Topic Embeddingscited by 36
  13. 2023Transformed Distribution Matching for Missing Value Imputationcited by 31
  14. 2023Vector Quantized Wasserstein Auto-Encodercited by 22
  15. 2015Gamma Belief Networkscited by 19
  16. 2021Learning Prototype-oriented Set Representations for Meta-Learningcited by 19
  17. 2023Contextualized Topic Coherence Metricscited by 19
  18. 2023Zero-Shot Cross-Lingual Summarization via Large Language Modelscited by 16
  19. 2018Dirichlet belief networks for topic structure learningcited by 12
  20. 2017An Automatic Approach for Document-level Topic Model Evaluationcited by 11
  21. 2013Supervised Topical Key Phrase Extraction of News Stories using Crowdsourcing, Light Filtering and Co-reference Normalizationcited by 10
  22. 2021Linguistic Dependencies and Statistical Dependencecited by 10
  23. 2023LLM-TAKE: Theme Aware Keyword Extraction Using Large Language Modelscited by 8
  24. 2024Optimal Transport for Structure Learning Under Missing Datacited by 5
  25. 2023Large Language Models as Zero-Shot Keyphrase Extractors: A Preliminary Empirical Studycited by 4
  26. 2023Towards Generalising Neural Topical Representationscited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 3; 2026: 7.

11 papers.

  1. 2025Bridging the Evaluation Gap: Leveraging Large Language Models for Topic Model Evaluationcited by 4
  2. 2025Toward Purpose-oriented Topic Model Evaluation enabled by Large Language Modelscited by 1
  3. 2026MALTopic: Multi-Agent LLM Topic Modeling Frameworkcited by 1
  4. 2024Iterative Improvement of an Additively Regularized Topic Modelcited by 0
  5. 2025Knowing Your Uncertainty -- On the application of LLM in social sciencescited by 0
  6. 2026Improving Topic Modeling by Distilling Soft Labels from Language Modelscited by 0
  7. 2026A Neural Topic Method Using a Large-Language-Model-in-the-Loop for Business Researchcited by 0
  8. 2026PRISM: PRIor from corpus Statistics for topic Modelingcited by 0
  9. 2026AnTenA: Actionable and Explainable Tensor Analysis System with Large Language Modelscited by 0
  10. 2026Advanced Topic Modeling Techniques for Categorizing Software Vulnerabilitiescited by 0
  11. 2026Document Topic Alignment Metrics for Evaluating Topic Models of Short-Text Public Health Communications on Social Mediacited by 0