# Statistical Agnostic Regression: a machine learning method to validate regression models

arXiv: https://arxiv.org/abs/2402.15213 · HTML: https://mlsanity.com/paper/2402.15213/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 9 of its 52 references that are in the map.
- First posted on arXiv 23 Feb 2024.

## Builds on (most cited first, 9 of 9)

- [The Shape of Learning Curves: a Review](https://mlsanity.com/paper/2103.10948/) (2021)
- [A PAC-Bayesian Tutorial with A Dropout Bound](https://mlsanity.com/paper/1307.2118/) (2013)
- [Cross-validation failure: small sample sizes lead to large error bars](https://mlsanity.com/paper/1706.07581/) (2017)
- [Can parametric statistical methods be trusted for fMRI based group studies?](https://mlsanity.com/paper/1511.01863/) (2015)
- [Is K-fold cross validation the best model selection method for Machine Learning?](https://mlsanity.com/paper/2401.16407/) (2024)
- [Better-Than-Chance Classification for Signal Detection](https://mlsanity.com/paper/1608.08873/) (2016)
- [Statistical Agnostic Mapping: a Framework in Neuroimaging based on Concentration Inequalities](https://mlsanity.com/paper/1912.12274/) (2019)
- [The Same Analysis Approach: Practical protection against the pitfalls of novel neuroimaging analysis methods](https://mlsanity.com/paper/1703.06670/) (2017)
- [A hypothesis-driven method based on machine learning for neuroimaging data analysis](https://mlsanity.com/paper/2202.04397/) (2022)

## Abstract (arXiv)

Regression analysis is a central topic in statistical modeling, aimed at estimating the relationships between a dependent variable, commonly referred to as the response variable, and one or more independent variables, i.e., explanatory variables. Linear regression is by far the most popular method for performing this task in various fields of research, such as data integration and predictive modeling when combining information from multiple sources. Classical methods for solving linear regression problems, such as Ordinary Least Squares (OLS), Ridge, or Lasso regressions, often form the foundation for more advanced machine learning (ML) techniques, which have been successfully applied, though without a formal definition of statistical significance. At most, permutation or analyses based on empirical measures (e.g., residuals or accuracy) have been conducted, leveraging the greater sensitivity of ML estimations for detection. In this paper, we introduce Statistical Agnostic Regression (SAR) for evaluating the statistical significance of ML-based linear regression models. This is achieved by analyzing concentration inequalities of the actual risk (expected loss) and considering the worst-case scenario. To this end, we define a threshold that ensures there is sufficient evidence, with a probability of at least 1-η, to conclude the existence of a linear relationship in the population between the explanatory (feature) and the response (label) variables. Simulations demonstrate the ability of the proposed agnostic (non-parametric) test to provide an analysis of variance similar to the classical multivariate F-test for the slope parameter, without relying on the underlying assumptions of classical methods. Moreover, the residuals computed from this method represent a trade-off between those obtained from ML approaches and the classical OLS.