# Likelihood estimation of sparse topic distributions in topic models and its applications to Wasserstein document distance calculations

arXiv: https://arxiv.org/abs/2107.05766 · HTML: https://mlsanity.com/paper/2107.05766/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 47 references that are in the map.
- First posted on arXiv 12 Jul 2021.

## Builds on (most cited first, 13 of 13)

- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Convex optimization](https://mlsanity.com/paper/2106.01946/) (2021)
- [Distributed Representations of Sentences and Documents](https://mlsanity.com/paper/1405.4053/) (2014)
- [Pre-trained Models for Natural Language Processing: A Survey](https://mlsanity.com/paper/2003.08271/) (2020)
- [An Introduction to Matrix Concentration Inequalities](https://mlsanity.com/paper/1501.01571/) (2015)
- [Inference for Empirical Wasserstein Distances on Finite Spaces](https://mlsanity.com/paper/1610.03287/) (2016)
- [Learning Markov models via low-rank optimization](https://mlsanity.com/paper/1907.00113/) (2019)
- [Provable Algorithms for Inference in Topic Models](https://mlsanity.com/paper/1605.08491/) (2016)
- [A fast algorithm with minimax optimal guarantees for topic models with an unknown number of topics](https://mlsanity.com/paper/1805.06837/) (2018)
- [Optimal estimation of sparse topic models](https://mlsanity.com/paper/2001.07861/) (2020)
- [A provable SVD-based algorithm for learning topics in dominant admixture corpus](https://mlsanity.com/paper/1410.6991/) (2014)
- [Topic Discovery through Data Dependent and Random Projections](https://mlsanity.com/paper/1303.3664/) (2013)

## Built on by (most cited first, 2 of 2)

- [Estimation and inference for the Wasserstein distance between mixing measures in topic models](https://mlsanity.com/paper/2206.12768/) (2022)
- [Fidelity-Diversity Metrics for Text](https://mlsanity.com/paper/2607.04563/) (2026)

## Abstract (arXiv)

This paper studies the estimation of high-dimensional, discrete, possibly sparse, mixture models in topic models. The data consists of observed multinomial counts of p words across n independent documents. In topic models, the p× n expected word frequency matrix is assumed to be factorized as a p× K word-topic matrix A and a K× n topic-document matrix T. Since columns of both matrices represent conditional probabilities belonging to probability simplices, columns of A are viewed as p-dimensional mixture components that are common to all documents while columns of T are viewed as the K-dimensional mixture weights that are document specific and are allowed to be sparse. The main interest is to provide sharp, finite sample, ℓ1-norm convergence rates for estimators of the mixture weights T when A is either known or unknown. For known A, we suggest MLE estimation of T. Our non-standard analysis of the MLE not only establishes its ℓ1 convergence rate, but reveals a remarkable property: the MLE, with no extra regularization, can be exactly sparse and contain the true zero pattern of T. We further show that the MLE is both minimax optimal and adaptive to the unknown sparsity in a large class of sparse topic distributions. When A is unknown, we estimate T by optimizing the likelihood function corresponding to a plug in, generic, estimator hatA of A. For any estimator hatA that satisfies carefully detailed conditions for proximity to A, the resulting estimator of T is shown to retain the properties established for the MLE. The ambient dimensions K and p are allowed to grow with the sample sizes. Our application is to the estimation of 1-Wasserstein distances between document generating distributions. We propose, estimate and analyze new 1-Wasserstein distances between two probabilistic document representations.