# Muddling Labels for Regularization, a novel approach to generalization

arXiv: https://arxiv.org/abs/2102.08769 · HTML: https://mlsanity.com/paper/2102.08769/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 60 references that are in the map.
- First posted on arXiv 17 Feb 2021.

## Builds on (most cited first, 16 of 16)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [Self-Normalizing Neural Networks](https://mlsanity.com/paper/1706.02515/) (2017)
- [Large-Scale Evolution of Image Classifiers](https://mlsanity.com/paper/1703.01041/) (2017)
- [Hyperparameter optimization with approximate gradient](https://mlsanity.com/paper/1602.02355/) (2016)
- [A disciplined approach to neural network hyper-parameters: Part 1 -- learning rate, batch size, momentum, and weight decay](https://mlsanity.com/paper/1803.09820/) (2018)
- [Regularization for Deep Learning: A Taxonomy](https://mlsanity.com/paper/1710.10686/) (2017)
- [Cross-validation failure: small sample sizes lead to large error bars](https://mlsanity.com/paper/1706.07581/) (2017)
- [Implicit differentiation of Lasso-type models for hyperparameter optimization](https://mlsanity.com/paper/2002.08943/) (2020)
- [Automating biomedical data science through tree-based pipeline optimization](https://mlsanity.com/paper/1601.07925/) (2016)
- [Autostacker: A Compositional Evolutionary Learning System](https://mlsanity.com/paper/1803.00684/) (2018)
- [Sequential Model-Based Ensemble Optimization](https://mlsanity.com/paper/1402.0796/) (2014)
- [Regularization Learning Networks: Deep Learning for Tabular Datasets](https://mlsanity.com/paper/1805.06440/) (2018)

## Built on by (most cited first, 1 of 1)

- [Muddling Label Regularization: Deep Learning for Tabular Datasets](https://mlsanity.com/paper/2106.04462/) (2021)

## Abstract (arXiv)

Generalization is a central problem in Machine Learning. Indeed most prediction methods require careful calibration of hyperparameters usually carried out on a hold-out textitvalidation dataset to achieve generalization. The main goal of this paper is to introduce a novel approach to achieve generalization without any data splitting, which is based on a new risk measure which directly quantifies a model's tendency to overfit. To fully understand the intuition and advantages of this new approach, we illustrate it in the simple linear regression model (Y=Xβ+xi) where we develop a new criterion. We highlight how this criterion is a good proxy for the true generalization risk. Next, we derive different procedures which tackle several structures simultaneously (correlation, sparsity,...). Noticeably, these procedures concomitantly train the model and calibrate the hyperparameters. In addition, these procedures can be implemented via classical gradient descent methods when the criterion is differentiable w.r.t. the hyperparameters. Our numerical experiments reveal that our procedures are computationally feasible and compare favorably to the popular approach (Ridge, LASSO and Elastic-Net combined with grid-search cross-validation) in term of generalization. They also outperform the baseline on two additional tasks: estimation and support recovery of β. Moreover, our procedures do not require any expertise for the calibration of the initial parameters which remain the same for all the datasets we experimented on.