# GRASP: A Goodness-of-Fit Test for Classification Learning

arXiv: https://arxiv.org/abs/2209.02064 · HTML: https://mlsanity.com/paper/2209.02064/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 77 references that are in the map.
- First posted on arXiv 5 Sep 2022.

## Builds on (most cited first, 19 of 19)

- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [f-GAN: Training Generative Neural Samplers using Variational Divergence Minimization](https://mlsanity.com/paper/1606.00709/) (2016)
- [Measuring Calibration in Deep Learning](https://mlsanity.com/paper/1904.01685/) (2019)
- [Verified Uncertainty Calibration](https://mlsanity.com/paper/1909.10155/) (2019)
- [Evaluating model calibration in classification](https://mlsanity.com/paper/1902.06977/) (2019)
- [Preserving Statistical Validity in Adaptive Data Analysis](https://mlsanity.com/paper/1411.2664/) (2014)
- [Controlling the false discovery rate via knockoffs](https://mlsanity.com/paper/1404.5609/) (2014)
- [Calibration tests in multi-class classification: A unifying framework](https://mlsanity.com/paper/1910.11385/) (2019)
- [Cross-validation: what does it estimate and how well does it do it?](https://mlsanity.com/paper/2104.00673/) (2021)
- [T-Cal: An optimal test for the calibration of predictive models](https://mlsanity.com/paper/2203.01850/) (2022)
- [Online Rules for Control of False Discovery Rate and False Discovery Exceedance](https://mlsanity.com/paper/1603.09000/) (2016)
- [Robust inference with knockoffs](https://mlsanity.com/paper/1801.03896/) (2018)
- [From Fixed-X to Random-X Regression: Bias-Variance Decompositions, Covariance Penalties, and Prediction Error Estimation](https://mlsanity.com/paper/1704.08160/) (2017)
- [Hypothesis Testing For Densities and High-Dimensional Multinomials: Sharp Local Minimax Rates](https://mlsanity.com/paper/1706.10003/) (2017)
- [Metropolized Knockoff Sampling](https://mlsanity.com/paper/1903.00434/) (2019)
- [Is a Classification Procedure Good Enough? A Goodness-of-Fit Assessment Tool for Classification Learning](https://mlsanity.com/paper/1911.03063/) (2019)
- [Goodness-of-fit testing in high-dimensional generalized linear models](https://mlsanity.com/paper/1908.03606/) (2019)
- [Goodness of fit tests for high-dimensional linear models](https://mlsanity.com/paper/1511.03334/) (2015)
- [Pearson Chi-squared Conditional Randomization Test](https://mlsanity.com/paper/2111.00027/) (2021)

## Built on by (most cited first, 1 of 1)

- [SPARK: A General Goodness-of-Fit Assessment via Residual Projection](https://mlsanity.com/paper/2609.37705/) (2026)

## Abstract (arXiv)

Performance of classifiers is often measured in terms of average accuracy on test data. Despite being a standard measure, average accuracy fails in characterizing the fit of the model to the underlying conditional law of labels given the features vector (Y|X), e.g. due to model misspecification, over fitting, and high-dimensionality. In this paper, we consider the fundamental problem of assessing the goodness-of-fit for a general binary classifier. Our framework does not make any parametric assumption on the conditional law Y|X, and treats that as a black box oracle model which can be accessed only through queries. We formulate the goodness-of-fit assessment problem as a tolerance hypothesis testing of the form [ H0: mathbbEBig[DfBig(sf Bern(η(X))|sf Bern(hatη(X))Big)Big]≤ τ,, ] where Df represents an f-divergence function, and η(x), hatη(x) respectively denote the true and an estimate likelihood for a feature vector x admitting a positive label. We propose a novel test, called grasp for testing H0, which works in finite sample settings, no matter the features (distribution-free). We also propose model-X grasp designed for model-X settings where the joint distribution of the features vector is known. Model-X grasp uses this distributional information to achieve better power. We evaluate the performance of our tests through extensive numerical experiments.