# Sparsity information and regularization in the horseshoe and other shrinkage priors

arXiv: https://arxiv.org/abs/1707.01694 · HTML: https://mlsanity.com/paper/1707.01694/

## Key facts

- Cited by 43 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Jul 2017.

## Built on by (most cited first, 30 of 43)

- [Pathfinder: Parallel quasi-Newton variational inference](https://mlsanity.com/paper/2108.03782/) (2021)
- [Amortized Bayesian Multilevel Models](https://mlsanity.com/paper/2408.13230/) (2024)
- [Gaussian processes meet NeuralODEs: A Bayesian framework for learning the dynamics of partially observed systems from scarce and noisy data](https://mlsanity.com/paper/2103.03385/) (2021)
- [Variational Inference in Location-Scale Families: Exact Recovery of the Mean and Correlation Matrix](https://mlsanity.com/paper/2410.11067/) (2024)
- [Adaptive Uncertainty-Guided Model Selection for Data-Driven PDE Discovery](https://mlsanity.com/paper/2308.10283/) (2023)
- [A fully Bayesian sparse polynomial chaos expansion approach with joint priors on the coefficients and global selection of terms](https://mlsanity.com/paper/2204.06043/) (2022)
- [Fourier Representations for Black-Box Optimization over Categorical Variables](https://mlsanity.com/paper/2202.03712/) (2022)
- [High-dimensional Bayesian Tobit regression for censored response with Horseshoe prior](https://mlsanity.com/paper/2505.08288/) (2025)
- [Quantifying sources of uncertainty in drug discovery predictions with probabilistic models](https://mlsanity.com/paper/2105.09474/) (2021)
- [The SKIM-FA Kernel: High-Dimensional Variable Selection and Nonlinear Interaction Discovery in Linear Time](https://mlsanity.com/paper/2106.12408/) (2021)
- [A fast asynchronous MCMC sampler for sparse Bayesian inference](https://mlsanity.com/paper/2108.06446/) (2021)
- [Encoding Domain Knowledge in Multi-view Latent Variable Models: A Bayesian Approach with Structured Sparsity](https://mlsanity.com/paper/2204.06242/) (2022)
- [Spatiotemporal Besov Priors for Bayesian Inverse Problems](https://mlsanity.com/paper/2306.16378/) (2023)
- [Optimal sparse phase retrieval via a quasi-Bayesian approach](https://mlsanity.com/paper/2504.09509/) (2025)
- [Generalized Guarantees for Variational Inference in the Presence of Even and Elliptical Symmetry](https://mlsanity.com/paper/2511.01064/) (2025)
- [Marginally calibrated response distributions for end-to-end learning in autonomous driving](https://mlsanity.com/paper/2110.01050/) (2021)
- [Reproducible Parameter Inference Using Bagged Posteriors](https://mlsanity.com/paper/2311.02019/) (2023)
- [Generative vs. Discriminative modeling under the lens of uncertainty quantification](https://mlsanity.com/paper/2406.09172/) (2024)
- [Handling bounded response in high dimensions: a Horseshoe prior Bayesian Beta regression approach](https://mlsanity.com/paper/2505.22211/) (2025)
- [Data-driven Discovery of Digital Twins in Biomedical Research](https://mlsanity.com/paper/2508.21484/) (2025)
- [Learning Physics between Digital Twins with Low-Fidelity Models and Physics-Informed Gaussian Processes](https://mlsanity.com/paper/2206.08201/) (2022)
- [A Targeted Accuracy Diagnostic for Variational Approximations](https://mlsanity.com/paper/2302.12419/) (2023)
- [Bayesian sparsification for deep neural networks with Bayesian model reduction](https://mlsanity.com/paper/2309.12095/) (2023)
- [Discovering Governing Equations in the Presence of Uncertainty](https://mlsanity.com/paper/2507.09740/) (2025)
- [Who and What? Using Linguistic Features and Annotator Characteristics to Analyze Annotation Variation](https://mlsanity.com/paper/2605.06318/) (2026)
- [Encoding Domain Information with Sparse Priors for Inferring Explainable Latent Variables](https://mlsanity.com/paper/2107.03730/) (2021)
- [Decoupling Shrinkage and Selection for the Bayesian Quantile Regression](https://mlsanity.com/paper/2107.08498/) (2021)
- [Interpretable Latent Variables in Deep State Space Models](https://mlsanity.com/paper/2203.02057/) (2022)
- [Autoencoded sparse Bayesian in-IRT factorization, calibration, and amortized inference for the Work Disability Functional Assessment Battery](https://mlsanity.com/paper/2210.10952/) (2022)
- [Streaming Bayesian Modeling for predicting Fat-Tailed Customer Lifetime Value](https://mlsanity.com/paper/2312.00373/) (2023)

## Abstract (arXiv)

The horseshoe prior has proven to be a noteworthy alternative for sparse Bayesian estimation, but has previously suffered from two problems. First, there has been no systematic way of specifying a prior for the global shrinkage hyperparameter based on the prior information about the degree of sparsity in the parameter vector. Second, the horseshoe prior has the undesired property that there is no possibility of specifying separately information about sparsity and the amount of regularization for the largest coefficients, which can be problematic with weakly identified parameters, such as the logistic regression coefficients in the case of data separation. This paper proposes solutions to both of these problems. We introduce a concept of effective number of nonzero parameters, show an intuitive way of formulating the prior for the global hyperparameter based on the sparsity assumptions, and argue that the previous default choices are dubious based on their tendency to favor solutions with more unshrunk parameters than we typically expect a priori. Moreover, we introduce a generalization to the horseshoe prior, called the regularized horseshoe, that allows us to specify a minimum level of regularization to the largest values. We show that the new prior can be considered as the continuous counterpart of the spike-and-slab prior with a finite slab width, whereas the original horseshoe resembles the spike-and-slab with an infinitely wide slab. Numerical experiments on synthetic and real world data illustrate the benefit of both of these theoretical advances.