# Easy Variational Inference for Categorical Models via an Independent Binary Approximation

arXiv: https://arxiv.org/abs/2206.00093 · HTML: https://mlsanity.com/paper/2206.00093/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 53 references that are in the map.
- First posted on arXiv 31 May 2022.

## Builds on (most cited first, 10 of 10)

- [Variational Inference: A Review for Statisticians](https://mlsanity.com/paper/1601.00670/) (2016)
- [Importance Weighted Autoencoders](https://mlsanity.com/paper/1509.00519/) (2015)
- [Automatic Differentiation Variational Inference](https://mlsanity.com/paper/1603.00788/) (2016)
- [Dependent Multinomial Models Made Easy: Stick Breaking with the P'olya-Gamma Augmentation](https://mlsanity.com/paper/1506.05843/) (2015)
- [Multi-Class Gaussian Process Classification Made Conjugate: Efficient Inference via Data Augmentation](https://mlsanity.com/paper/1905.09670/) (2019)
- [Scalable and Accurate Variational Bayes for High-Dimensional Binary Regression Models](https://mlsanity.com/paper/1911.06743/) (2019)
- [Conditionally conjugate mean-field variational Bayes for logistic models](https://mlsanity.com/paper/1711.06999/) (2017)
- [Bayesian Non-Homogeneous Markov Models via Polya-Gamma Data Augmentation with Applications to Rainfall Modeling](https://mlsanity.com/paper/1701.02856/) (2017)
- [Permuted and Augmented Stick-Breaking Bayesian Multinomial Regression](https://mlsanity.com/paper/1612.09413/) (2016)
- [A Symmetric Prior for Multinomial Probit Models](https://mlsanity.com/paper/1912.10334/) (2019)

## Built on by (most cited first, 2 of 2)

- [Streaming Bayes GFlowNets](https://mlsanity.com/paper/2411.05899/) (2024)
- [Variational Filtering for Bayesian Last Layers with an Independent Binary Probit Surrogate](https://mlsanity.com/paper/2610.16578/) (2026)

## Abstract (arXiv)

We pursue tractable Bayesian analysis of generalized linear models (GLMs) for categorical data. Thus far, GLMs are difficult to scale to more than a few dozen categories due to non-conjugacy or strong posterior dependencies when using conjugate auxiliary variable methods. We define a new class of GLMs for categorical data called categorical-from-binary (CB) models. Each CB model has a likelihood that is bounded by the product of binary likelihoods, suggesting a natural posterior approximation. This approximation makes inference straightforward and fast; using well-known auxiliary variables for probit or logistic regression, the product of binary models admits conjugate closed-form variational inference that is embarrassingly parallel across categories and invariant to category ordering. Moreover, an independent binary model simultaneously approximates multiple CB models. Bayesian model averaging over these can improve the quality of the approximation for any given dataset. We show that our approach scales to thousands of categories, outperforming posterior estimation competitors like Automatic Differentiation Variational Inference (ADVI) and No U-Turn Sampling (NUTS) in the time required to achieve fixed prediction quality.