Repeatability is not recovery: Quantifying algorithmic stability and topic recovery in Latent Dirichlet Allocation

Topic models are often judged by the consistency of their outputs across repeated runs, implicitly assuming that repeatable topic output is a successful recovery of the underlying topics. We show that this assumption is false: repeatability is not recovery. We introduce a stability framework that jointly measures consistency among repeated runs and accuracy relative to known ground truth. Because real-world corpora lack known topic structures, we generate synthetic corpora using the Latent Dirichlet Allocation (LDA) generative process, enabling direct evaluation of topic recovery. Across 50 repeated LDA runs on each corpus, we find that LDA reliably identifies the correct number of topics and frequently converges to highly consistent topic solutions. However, these repeatable solutions frequently fail to recover the true generating topics. Thus, internal stability should not be interpreted as evidence of correctness. Our results illustrate that stability and recovery are distinct properties of topic models and should be evaluated separately. Consequently, topic-model outputs should be validated using multiple complementary criteria before supporting substantive conclusions, particularly in high-stakes applications.