# A new evaluation framework for topic modeling algorithms based on synthetic corpora

arXiv: https://arxiv.org/abs/1901.09848 · HTML: https://mlsanity.com/paper/1901.09848/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Jan 2019.

## Built on by (most cited first, 3 of 3)

- [Why the Rich Get Richer? On the Balancedness of Random Partition Models](https://mlsanity.com/paper/2201.12697/) (2022)
- [Lumen: A Machine Learning Framework to Expose Influence Cues in Text](https://mlsanity.com/paper/2107.10655/) (2021)
- [LLM-Generated Negative News Headlines Dataset: Creation and Benchmarking Against Real Journalism](https://mlsanity.com/paper/2511.11591/) (2025)

## Abstract (arXiv)

Topic models are in widespread use in natural language processing and beyond. Here, we propose a new framework for the evaluation of probabilistic topic modeling algorithms based on synthetic corpora containing an unambiguously defined ground truth topic structure. The major innovation of our approach is the ability to quantify the agreement between the planted and inferred topic structures by comparing the assigned topic labels at the level of the tokens. In experiments, our approach yields novel insights about the relative strengths of topic models as corpus characteristics vary, and the first evidence of an "undetectable phase" for topic models when the planted structure is weak. We also establish the practical relevance of the insights gained for synthetic corpora by predicting the performance of topic modeling algorithms in classification tasks in real-world corpora.