# Hierarchical mixtures of Unigram models for short text clustering: The role of Beta-Liouville priors

arXiv: https://arxiv.org/abs/2410.21862 · HTML: https://mlsanity.com/paper/2410.21862/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 7 of its 57 references that are in the map.
- First posted on arXiv 29 Oct 2024.

## Builds on (most cited first, 7 of 7)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Variational Inference: A Review for Statisticians](https://mlsanity.com/paper/1601.00670/) (2016)
- [Optimization Methods for Large-Scale Machine Learning](https://mlsanity.com/paper/1606.04838/) (2016)
- [Advances in Variational Inference](https://mlsanity.com/paper/1711.05597/) (2017)
- [A Survey on Neural Topic Models: Methods, Applications, and Challenges](https://mlsanity.com/paper/2401.15351/) (2024)
- [A practical tutorial on Variational Bayes](https://mlsanity.com/paper/2103.01327/) (2021)
- [Dealing with overdispersion in multivariate count data](https://mlsanity.com/paper/2107.00470/) (2021)

## Abstract (arXiv)

This paper presents a variant of the Multinomial mixture model tailored to the unsupervised classification of short text data. While the Multinomial probability vector is traditionally assigned a Dirichlet prior distribution, this work explores an alternative formulation based on the Beta-Liouville distribution, which offers a more flexible correlation structure than the Dirichlet. We examine the theoretical properties of the Beta-Liouville distribution, with particular focus on its conjugacy with the Multinomial likelihood. This property enables the derivation of update equations for a CAVI (Coordinate Ascent Variational Inference) algorithm, facilitating approximate posterior inference of the model parameters. In addition, we introduce a stochastic variant of the CAVI algorithm to enhance scalability. The paper concludes with empirical examples demonstrating effective strategies for selecting the Beta-Liouville hyperparameters.