# Entropy regularization in probabilistic clustering

arXiv: https://arxiv.org/abs/2307.10065 · HTML: https://mlsanity.com/paper/2307.10065/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 28 of its 76 references that are in the map.
- First posted on arXiv 19 Jul 2023.

## Builds on (most cited first, 28 of 28)

- [Mixture models with a prior on the number of components](https://mlsanity.com/paper/1502.06241/) (2015)
- [On the nonparametric maximum likelihood estimator for Gaussian location mixture densities with application to Gaussian denoising](https://mlsanity.com/paper/1712.02009/) (2017)
- [Self-regularizing Property of Nonparametric Maximum Likelihood Estimator in Mixture Models](https://mlsanity.com/paper/2008.08244/) (2020)
- [Bayesian cluster analysis: Point estimation and credible balls](https://mlsanity.com/paper/1505.03339/) (2015)
- [Are Gibbs-type priors the most natural generalization of the Dirichlet process?](https://mlsanity.com/paper/1503.00163/) (2015)
- [Flexible Models for Microclustering with Application to Entity Resolution](https://mlsanity.com/paper/1610.09780/) (2016)
- [Search Algorithms and Loss Functions for Bayesian Clustering](https://mlsanity.com/paper/2105.04451/) (2021)
- [Bayesian Repulsive Gaussian Mixture Model](https://mlsanity.com/paper/1703.09061/) (2017)
- [Random Partition Models for Microclustering Tasks](https://mlsanity.com/paper/2004.02008/) (2020)
- [Clustering consistency with Dirichlet process mixtures](https://mlsanity.com/paper/2205.12924/) (2022)
- [Optimal Bayesian estimators for latent variable cluster models](https://mlsanity.com/paper/1607.02325/) (2016)
- [Latent nested nonparametric priors](https://mlsanity.com/paper/1801.05048/) (2018)
- [Flexible clustering via hidden hierarchical Dirichlet priors](https://mlsanity.com/paper/2201.06994/) (2022)
- [Centered Partition Process: Informative Priors for Clustering](https://mlsanity.com/paper/1901.10225/) (2019)
- [A Common Atom Model for the Bayesian Nonparametric Analysis of Nested Data](https://mlsanity.com/paper/2008.07077/) (2020)
- [Determinantal point process mixtures via spectral density approach](https://mlsanity.com/paper/1705.05181/) (2017)
- [Is infinity that far? A Bayesian nonparametric perspective of finite mixture models](https://mlsanity.com/paper/1904.09733/) (2019)
- [Dependent Modeling of Temporal Sequences of Random Partitions](https://mlsanity.com/paper/1912.11542/) (2019)
- [The semi-hierarchical Dirichlet Process and its application to clustering homogeneous distributions](https://mlsanity.com/paper/2005.10287/) (2020)
- [MCMC computations for Bayesian mixture models using repulsive point processes](https://mlsanity.com/paper/2011.06444/) (2020)
- [Spying on the prior of the number of data clusters and the partition distribution in Bayesian cluster analysis](https://mlsanity.com/paper/2012.12337/) (2020)
- [Separate Exchangeability as Modeling Principle in Bayesian Nonparametrics](https://mlsanity.com/paper/2112.07755/) (2021)
- [Why the Rich Get Richer? On the Balancedness of Random Partition Models](https://mlsanity.com/paper/2201.12697/) (2022)
- [Bayesian clustering of multiple zero-inflated outcomes](https://mlsanity.com/paper/2205.05054/) (2022)
- [Beta-Binomial stick-breaking non-parametric prior](https://mlsanity.com/paper/1908.06602/) (2019)
- [Clustering Areal Units at Multiple Levels of Resolution to Model Crime in Philadelphia](https://mlsanity.com/paper/2112.02059/) (2021)
- [Model Selection for Maternal Hypertensive Disorders with Symmetric Hierarchical Dirichlet Processes](https://mlsanity.com/paper/2203.15782/) (2022)
- [Graph-Aligned Random Partition Model (GARP)](https://mlsanity.com/paper/2306.08485/) (2023)

## Abstract (arXiv)

Bayesian nonparametric mixture models are widely used to cluster observations. However, one major drawback of the approach is that the estimated partition often presents unbalanced clusters' frequencies with only a few dominating clusters and a large number of sparsely-populated ones. This feature translates into results that are often uninterpretable unless we accept to ignore a relevant number of observations and clusters. Interpreting the posterior distribution as penalized likelihood, we show how the unbalance can be explained as a direct consequence of the cost functions involved in estimating the partition. In light of our findings, we propose a novel Bayesian estimator of the clustering configuration. The proposed estimator is equivalent to a post-processing procedure that reduces the number of sparsely-populated clusters and enhances interpretability. The procedure takes the form of entropy-regularization of the Bayesian estimate. While being computationally convenient with respect to alternative strategies, it is also theoretically justified as a correction to the Bayesian loss function used for point estimation and, as such, can be applied to any posterior distribution of clusters, regardless of the specific model used.