# A Flexible Empirical Bayes Approach to Generalized Linear Models, with Applications to Sparse Logistic Regression

arXiv: https://arxiv.org/abs/2601.21217 · HTML: https://mlsanity.com/paper/2601.21217/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 18 of its 55 references that are in the map.
- First posted on arXiv 29 Jan 2026.

## Builds on (most cited first, 18 of 18)

- [Convex optimization](https://mlsanity.com/paper/2106.01946/) (2021)
- [Variational Inference: A Review for Statisticians](https://mlsanity.com/paper/1601.00670/) (2016)
- [Black Box Variational Inference](https://mlsanity.com/paper/1401.0118/) (2013)
- [Automatic Differentiation Variational Inference](https://mlsanity.com/paper/1603.00788/) (2016)
- [Sparsity information and regularization in the horseshoe and other shrinkage priors](https://mlsanity.com/paper/1707.01694/) (2017)
- [Pathfinder: Parallel quasi-Newton variational inference](https://mlsanity.com/paper/2108.03782/) (2021)
- [Fast Sparse Classification for Generalized Linear and Additive Models](https://mlsanity.com/paper/2202.11389/) (2022)
- [Two Modeling Strategies for Empirical Bayes Estimation](https://mlsanity.com/paper/1409.2677/) (2014)
- [Spike and slab variational Bayes for high dimensional logistic regression](https://mlsanity.com/paper/2010.11665/) (2020)
- [Beyond L1: Faster and Better Sparse Models with skglm](https://mlsanity.com/paper/2204.07826/) (2022)
- [L0Learn: A Scalable Package for Sparse Learning using L0 Regularization](https://mlsanity.com/paper/2202.04820/) (2022)
- [A flexible empirical Bayes approach to multiple linear regression and connections with penalized regression](https://mlsanity.com/paper/2208.10910/) (2022)
- [Understanding Stochastic Natural Gradient Variational Inference](https://mlsanity.com/paper/2406.01870/) (2024)
- [Scalable Spike-and-Slab](https://mlsanity.com/paper/2204.01668/) (2022)
- [Variational Gaussian Approximation for Poisson Data](https://mlsanity.com/paper/1709.05885/) (2017)
- [ebnm: An R Package for Solving the Empirical Bayes Normal Means Problem Using a Variety of Prior Families](https://mlsanity.com/paper/2110.00152/) (2021)
- [Intuitive Joint Priors for Bayesian Linear Multilevel Models: The R2D2M2 prior](https://mlsanity.com/paper/2208.07132/) (2022)
- [yaglm: a Python package for fitting and tuning generalized linear models that supports structured, adaptive and non-convex penalties](https://mlsanity.com/paper/2110.05567/) (2021)

## Abstract (arXiv)

We introduce a flexible empirical Bayes approach for fitting Bayesian generalized linear models. Specifically, we adopt a novel mean-field variational inference (VI) method and the prior is estimated within the VI algorithm, making the method tuning-free. Unlike traditional VI methods that optimize the posterior density function, our approach directly optimizes the posterior mean and prior parameters. This formulation reduces the number of parameters to optimize and enables the use of scalable algorithms such as L-BFGS and stochastic gradient descent. Furthermore, our method automatically determines the optimal posterior based on the prior and likelihood, distinguishing it from existing VI methods that often assume a Gaussian variational. Our approach represents a unified framework applicable to a wide range of exponential family distributions, removing the need to develop unique VI methods for each combination of likelihood and prior distributions. We apply the framework to solve sparse logistic regression and demonstrate the superior predictive performance of our method in extensive numerical studies, by comparing it to prevalent sparse logistic regression approaches.