# Using SVD for Topic Modeling

arXiv: https://arxiv.org/abs/1704.07016 · HTML: https://mlsanity.com/paper/1704.07016/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Apr 2017.

## Built on by (most cited first, 14 of 14)

- [Freeze then Train: Towards Provable Representation Learning under Spurious Correlations and Feature Noise](https://mlsanity.com/paper/2210.11075/) (2022)
- [Estimation and inference for the Wasserstein distance between mixing measures in topic models](https://mlsanity.com/paper/2206.12768/) (2022)
- [Improved Algorithm and Bounds for Successive Projection](https://mlsanity.com/paper/2403.11013/) (2024)
- [Mixed membership estimation for categorical data with weighted responses](https://mlsanity.com/paper/2310.10989/) (2023)
- [Fundamental Limits of Learning High-dimensional Simplices in Noisy Regimes](https://mlsanity.com/paper/2506.10101/) (2025)
- [Poisson-Process Topic Model for Integrating Knowledge from Pre-trained Language Models](https://mlsanity.com/paper/2503.17809/) (2025)
- [Factor Augmented Supervised Learning with Text Embeddings](https://mlsanity.com/paper/2508.06548/) (2025)
- [Estimating mixed memberships in multi-layer networks](https://mlsanity.com/paper/2404.03916/) (2024)
- [Generalized Grade-of-Membership Estimation for High-dimensional Locally Dependent Data](https://mlsanity.com/paper/2412.19796/) (2024)
- [Fitting networks with a cancellation trick](https://mlsanity.com/paper/2502.16728/) (2025)
- [Semi-supervised Vertex Hunting, with Applications in Network and Text Analysis](https://mlsanity.com/paper/2510.22526/) (2025)
- [Dynamic Topic Modeling with a Higher-Order Hypergraphical Representation](https://mlsanity.com/paper/2605.28269/) (2026)
- [Multiclass Classification without Labels via Posterior Simplex Geometry](https://mlsanity.com/paper/2607.24943/) (2026)
- [Scalable Minimum-Volume Simplex Estimation with Non-asymptotic Analysis](https://mlsanity.com/paper/2609.25576/) (2026)

## Abstract (arXiv)

The probabilistic topic model imposes a low-rank structure on the expectation of the corpus matrix. Therefore, singular value decomposition (SVD) is a natural tool of dimension reduction. We propose an SVD-based method for estimating a topic model. Our method constructs an estimate of the topic matrix from only a few leading singular vectors of the corpus matrix, and has a great advantage in memory use and computational cost for large-scale corpora. The core ideas behind our method include a pre-SVD normalization to tackle severe word frequency heterogeneity, a post-SVD normalization to create a low-dimensional word embedding that manifests a simplex geometry, and a post-SVD procedure to construct an estimate of the topic matrix directly from the embedded word cloud. We provide the explicit rate of convergence of our method. We show that our method attains the optimal rate in the case of long and moderately long documents, and it improves the rates of existing methods in the case of short documents. The key of our analysis is a sharp row-wise large-deviation bound for empirical singular vectors, which is technically demanding to derive and potentially useful for other problems. We apply our method to a corpus of Associated Press news articles and a corpus of abstracts of statistical papers.