# Nonparametric Hypothesis Testing of High-dimensional Clustering With Application to Single-cell RNA Data

arXiv: https://arxiv.org/abs/2609.05683 · HTML: https://mlsanity.com/paper/2609.05683/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 13 of its 43 references that are in the map.
- First posted on arXiv 4 Sep 2026.

## Builds on (most cited first, 13 of 13)

- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Input Convex Neural Networks](https://mlsanity.com/paper/1609.07152/) (2016)
- [Sliced Score Matching: A Scalable Approach to Density and Score Estimation](https://mlsanity.com/paper/1905.07088/) (2019)
- [Theoretical guarantees for approximate sampling from smooth and log-concave densities](https://mlsanity.com/paper/1412.7392/) (2014)
- [Selective Inference for Hierarchical Clustering](https://mlsanity.com/paper/2012.02936/) (2020)
- [Selective inference for k-means clustering](https://mlsanity.com/paper/2203.15267/) (2022)
- [Recent progress in log-concave density estimation](https://mlsanity.com/paper/1709.03154/) (2017)
- [Statistical Significance of Clustering using Soft Thresholding](https://mlsanity.com/paper/1305.5879/) (2013)
- [High-dimensional nonparametric density estimation via symmetry and shape constraints](https://mlsanity.com/paper/1903.06092/) (2019)
- [Exact Cluster Recovery via Classical Multidimensional Scaling](https://mlsanity.com/paper/1812.11954/) (2018)
- [A Polynomial Time Algorithm for Log-Concave Maximum Likelihood via Locally Exponential Families](https://mlsanity.com/paper/1907.08306/) (2019)
- [Gaussian Mixture Clustering Using Relative Tests of Fit](https://mlsanity.com/paper/1910.02566/) (2019)
- [Fast Multivariate Log-Concave Density Estimation](https://mlsanity.com/paper/1805.07272/) (2018)

## Abstract (arXiv)

Single-cell RNA sequencing studies routinely use clustering to define putative cell types and cell states, yet the observed separation may arise from sampling variability rather than genuine biological heterogeneity. This paper studies formal significance testing of such clustering structure in high-dimensional data. Existing SigClust methods assess clustering significance through Monte Carlo simulation under a Gaussian single-cluster null, but this assumption can be unreliable for normalized gene expression data and other non-Gaussian settings. We propose SigClust-LCP, a nonparametric extension that models a single cluster by a log-concave distribution. To make this approach computationally feasible in moderate to high dimensions, we develop a score-matching estimator for log-concave projection inspired by recent generative modeling ideas. We establish theoretical guarantees for the estimator and for its use in clustering significance testing. Simulations show that SigClust-LCP controls Type-I error more reliably than existing methods across a range of unimodal and mixture distributions while retaining competitive power. In a single-cell RNA sequencing analysis of Hydra cells, the method avoids spurious subclusters within annotated cell populations and supports biologically meaningful separation across lineages and body-axis regions.