# Reliability of Topic Modeling

arXiv: https://arxiv.org/abs/2410.23186 · HTML: https://mlsanity.com/paper/2410.23186/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 61 references that are in the map.
- First posted on arXiv 30 Oct 2024.

## Builds on (most cited first, 8 of 8)

- [BERTopic: Neural topic modeling with a class-based TF-IDF procedure](https://mlsanity.com/paper/2203.05794/) (2022)
- [Is Automated Topic Model Evaluation Broken?: The Incoherence of Coherence](https://mlsanity.com/paper/2107.02173/) (2021)
- [Evaluating Evaluation Metrics: A Framework for Analyzing NLG Evaluation Metrics using Measurement Theory](https://mlsanity.com/paper/2305.14889/) (2023)
- [Reliability Testing for Natural Language Processing Systems](https://mlsanity.com/paper/2105.02590/) (2021)
- [Assessing the Reliability of Word Embedding Gender Bias Measures](https://mlsanity.com/paper/2109.04732/) (2021)
- [Generating Synthetic Text Data to Evaluate Causal Inference Methods](https://mlsanity.com/paper/2102.05638/) (2021)
- [Latent Dirichlet Allocation (LDA) for Topic Modeling of the CFPB Consumer Complaints](https://mlsanity.com/paper/1807.07468/) (2018)
- [Predicting Embedding Reliability in Low-Resource Settings Using Corpus Similarity Measures](https://mlsanity.com/paper/2206.04330/) (2022)

## Built on by (most cited first, 1 of 1)

- [From Complaint Narratives to Monetary Relief: A Hybrid Machine Learning Framework for CFPB Consumer Complaints](https://mlsanity.com/paper/2606.22664/) (2026)

## Abstract (arXiv)

Topic models allow researchers to extract latent factors from text data and use those variables in downstream statistical analyses. However, these methodologies can vary significantly due to initialization differences, randomness in sampling procedures, or noisy data. Reliability of these methods is of particular concern as many researchers treat learned topic models as ground truth for subsequent analyses. In this work, we show that the standard practice for quantifying topic model reliability fails to capture essential aspects of the variation in two widely-used topic models. Drawing from a extensive literature on measurement theory, we provide empirical and theoretical analyses of three other metrics for evaluating the reliability of topic models. On synthetic and real-world data, we show that McDonald's ω provides the best encapsulation of reliability. This metric provides an essential tool for validation of topic model methodologies that should be a standard component of any topic model-based research.