# Is a Classification Procedure Good Enough? A Goodness-of-Fit Assessment Tool for Classification Learning

arXiv: https://arxiv.org/abs/1911.03063 · HTML: https://mlsanity.com/paper/1911.03063/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Nov 2019.

## Built on by (most cited first, 9 of 9)

- [Federated Learning Challenges and Opportunities: An Outlook](https://mlsanity.com/paper/2202.00807/) (2022)
- [Is this model reliable for everyone? Testing for strong calibration](https://mlsanity.com/paper/2307.15247/) (2023)
- [From Conditional to Unconditional Independence: Testing Conditional Independence via Transport Maps](https://mlsanity.com/paper/2504.09567/) (2025)
- [GRASP: A Goodness-of-Fit Test for Classification Learning](https://mlsanity.com/paper/2209.02064/) (2022)
- [On the Role of Calibration in Benchmarking Algorithmic Fairness for Skin Cancer Detection](https://mlsanity.com/paper/2511.07700/) (2025)
- [Hierarchical Neyman-Pearson Classification for Prioritizing Severe Disease Categories in COVID-19 Patient Data](https://mlsanity.com/paper/2210.02197/) (2022)
- [Machine-Learning-Assisted Comparison of Regression Functions](https://mlsanity.com/paper/2510.24714/) (2025)
- [[EDGE] A grouped calibration test for logistic regression that tolerates a few corrupted records](https://mlsanity.com/paper/2608.20511/) (2026)
- [SPARK: A General Goodness-of-Fit Assessment via Residual Projection](https://mlsanity.com/paper/2609.37705/) (2026)

## Abstract (arXiv)

In recent years, many non-traditional classification methods, such as Random Forest, Boosting, and neural network, have been widely used in applications. Their performance is typically measured in terms of classification accuracy. While the classification error rate and the like are important, they do not address a fundamental question: Is the classification method underfitted? To our best knowledge, there is no existing method that can assess the goodness-of-fit of a general classification procedure. Indeed, the lack of a parametric assumption makes it challenging to construct proper tests. To overcome this difficulty, we propose a methodology called BAGofT that splits the data into a training set and a validation set. First, the classification procedure to assess is applied to the training set, which is also used to adaptively find a data grouping that reveals the most severe regions of underfitting. Then, based on this grouping, we calculate a test statistic by comparing the estimated success probabilities and the actual observed responses from the validation set. The data splitting guarantees that the size of the test is controlled under the null hypothesis, and the power of the test goes to one as the sample size increases under the alternative hypothesis. For testing parametric classification models, the BAGofT has a broader scope than the existing methods since it is not restricted to specific parametric models (e.g., logistic regression). Extensive simulation studies show the utility of the BAGofT when assessing general classification procedures and its strengths over some existing methods when testing parametric classification models.