# Nonparametric Empirical Bayes Estimation and Testing for Sparse and Heteroscedastic Signals

arXiv: https://arxiv.org/abs/2106.08881 · HTML: https://mlsanity.com/paper/2106.08881/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 9 of its 70 references that are in the map.
- First posted on arXiv 16 Jun 2021.

## Builds on (most cited first, 9 of 9)

- [Personalized HeartSteps: A Reinforcement Learning Algorithm for Optimizing Physical Activity](https://mlsanity.com/paper/1909.03539/) (2019)
- [Two Modeling Strategies for Empirical Bayes Estimation](https://mlsanity.com/paper/1409.2677/) (2014)
- [AdaPT: An interactive procedure for multiple testing with side information](https://mlsanity.com/paper/1609.06035/) (2016)
- [Adaptive multigroup confidence intervals with constant coverage](https://mlsanity.com/paper/1612.08287/) (2016)
- [Group-Linear Empirical Bayes Estimates for a Heteroscedastic Normal Mean](https://mlsanity.com/paper/1503.08503/) (2015)
- [Heterocedasticity-Adjusted Ranking and Thresholding for Large-Scale Multiple Testing](https://mlsanity.com/paper/1910.08107/) (2019)
- [Nonparametric Empirical Bayes Estimation on Heterogeneous Data](https://mlsanity.com/paper/2002.12586/) (2020)
- [A Decision Theoretic Approach to A/B Testing](https://mlsanity.com/paper/1710.03410/) (2017)
- [Optimal post-selection inference for sparse signals: a nonparametric empirical-Bayes approach](https://mlsanity.com/paper/1810.11042/) (2018)

## Abstract (arXiv)

Large-scale modern data often involves estimation and testing for high-dimensional unknown parameters. It is desirable to identify the sparse signals, ``the needles in the haystack'', with accuracy and false discovery control. However, the unprecedented complexity and heterogeneity in modern data structure require new machine learning tools to effectively exploit commonalities and to robustly adjust for both sparsity and heterogeneity. In addition, estimates for high-dimensional parameters often lack uncertainty quantification. In this paper, we propose a novel Spike-and-Nonparametric mixture prior (SNP) -- a spike to promote the sparsity and a nonparametric structure to capture signals. In contrast to the state-of-the-art methods, the proposed methods solve the estimation and testing problem at once with several merits: 1) an accurate sparsity estimation; 2) point estimates with shrinkage/soft-thresholding property; 3) credible intervals for uncertainty quantification; 4) an optimal multiple testing procedure that controls false discovery rate. Our method exhibits promising empirical performance on both simulated data and a gene expression case study.