# RATT: Leveraging Unlabeled Data to Guarantee Generalization

arXiv: https://arxiv.org/abs/2105.00303 · HTML: https://mlsanity.com/paper/2105.00303/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 51 of its 86 references that are in the map.
- First posted on arXiv 1 May 2021.

## Builds on (most cited first, 30 of 51)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Neural Tangent Kernel: Convergence and Generalization in Neural Networks](https://mlsanity.com/paper/1806.07572/) (2018)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [Understanding deep learning requires rethinking generalization](https://mlsanity.com/paper/1611.03530/) (2016)
- [Do ImageNet Classifiers Generalize to ImageNet?](https://mlsanity.com/paper/1902.10811/) (2019)
- [A Closer Look at Memorization in Deep Networks](https://mlsanity.com/paper/1706.05394/) (2017)
- [A Convergence Theory for Deep Learning via Over-Parameterization](https://mlsanity.com/paper/1811.03962/) (2018)
- [Spectrally-normalized margin bounds for neural networks](https://mlsanity.com/paper/1706.08498/) (2017)
- [On Lazy Training in Differentiable Programming](https://mlsanity.com/paper/1812.07956/) (2018)
- [Gradient Descent Provably Optimizes Over-parameterized Neural Networks](https://mlsanity.com/paper/1810.02054/) (2018)
- [The Implicit Bias of Gradient Descent on Separable Data](https://mlsanity.com/paper/1710.10345/) (2017)
- [Exploring Generalization in Deep Learning](https://mlsanity.com/paper/1706.08947/) (2017)
- [Gradient Descent Finds Global Minima of Deep Neural Networks](https://mlsanity.com/paper/1811.03804/) (2018)
- [Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networks](https://mlsanity.com/paper/1901.08584/) (2019)
- [Train faster, generalize better: Stability of stochastic gradient descent](https://mlsanity.com/paper/1509.01240/) (2015)
- [Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Data](https://mlsanity.com/paper/1703.11008/) (2017)
- [Early-Learning Regularization Prevents Memorization of Noisy Labels](https://mlsanity.com/paper/2007.00151/) (2020)
- [Learning and Generalization in Overparameterized Neural Networks, Going Beyond Two Layers](https://mlsanity.com/paper/1811.04918/) (2018)
- [Implicit Regularization in Matrix Factorization](https://mlsanity.com/paper/1705.09280/) (2017)
- [A PAC-Bayesian Approach to Spectrally-Normalized Margin Bounds for Neural Networks](https://mlsanity.com/paper/1707.09564/) (2017)
- [Stronger generalization bounds for deep nets via a compression approach](https://mlsanity.com/paper/1802.05296/) (2018)
- [Information-theoretic analysis of generalization capability of learning algorithms](https://mlsanity.com/paper/1705.07809/) (2017)
- [Norm-Based Capacity Control in Neural Networks](https://mlsanity.com/paper/1503.00036/) (2015)
- [Implicit Bias of Gradient Descent for Wide Two-layer Neural Networks Trained with the Logistic Loss](https://mlsanity.com/paper/2002.04486/) (2020)
- [Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Data](https://mlsanity.com/paper/1808.01204/) (2018)
- [Implicit Bias of Gradient Descent on Linear Convolutional Networks](https://mlsanity.com/paper/1806.00468/) (2018)
- [Uniform convergence may be unable to explain generalization in deep learning](https://mlsanity.com/paper/1902.04742/) (2019)
- [Gradient Descent Happens in a Tiny Subspace](https://mlsanity.com/paper/1812.04754/) (2018)

## Built on by (most cited first, 20 of 20)

- [Leveraging Unlabeled Data to Predict Out-of-Distribution Performance](https://mlsanity.com/paper/2201.04234/) (2022)
- [Assessing Generalization of SGD via Disagreement](https://mlsanity.com/paper/2106.13799/) (2021)
- [Agreement-on-the-Line: Predicting the Performance of Neural Networks under Distribution Shift](https://mlsanity.com/paper/2206.13089/) (2022)
- [Characterizing Datapoints via Second-Split Forgetting](https://mlsanity.com/paper/2210.15031/) (2022)
- [Domain Adaptation under Open Set Label Shift](https://mlsanity.com/paper/2207.13048/) (2022)
- [Predicting is not Understanding: Recognizing and Addressing Underspecification in Machine Learning](https://mlsanity.com/paper/2207.02598/) (2022)
- [On the Strong Correlation Between Model Invariance and Generalization](https://mlsanity.com/paper/2207.07065/) (2022)
- [Early Stopping Against Label Noise Without Validation Data](https://mlsanity.com/paper/2502.07551/) (2025)
- [(Almost) Provable Error Bounds Under Distribution Shift via Disagreement Discrepancy](https://mlsanity.com/paper/2306.00312/) (2023)
- [Explanation Shift: Detecting distribution shifts on tabular data via the explanation space](https://mlsanity.com/paper/2210.12369/) (2022)
- [Explanation Shift: How Did the Distribution Shift Impact the Model?](https://mlsanity.com/paper/2303.08081/) (2023)
- [Bounding generalization error with input compression: An empirical study with infinite-width networks](https://mlsanity.com/paper/2207.09408/) (2022)
- [Why the pseudo label based semi-supervised learning algorithm is effective?](https://mlsanity.com/paper/2211.10039/) (2022)
- [Beyond Demographic Parity: Redefining Equal Treatment](https://mlsanity.com/paper/2303.08040/) (2023)
- [Leveraging Unlabeled Data to Track Memorization](https://mlsanity.com/paper/2212.04461/) (2022)
- [Exploiting All Samples in Low-Resource Sentence Classification: Early Stopping and Initialization Parameters](https://mlsanity.com/paper/2111.06971/) (2021)
- [t-RAIN: Robust generalization under weather-aliasing label shift attacks](https://mlsanity.com/paper/2305.08302/) (2023)
- [Assessing Model Generalization in Vicinity](https://mlsanity.com/paper/2406.09257/) (2024)
- [Learning with Noisy Ground Truth: From 2D Classification to 3D Reconstruction](https://mlsanity.com/paper/2406.15982/) (2024)
- [Is Supervised Learning Really That Different from Unsupervised?](https://mlsanity.com/paper/2505.11006/) (2025)

## Abstract (arXiv)

To assess generalization, machine learning scientists typically either (i) bound the generalization gap and then (after training) plug in the empirical risk to obtain a bound on the true risk; or (ii) validate empirically on holdout data. However, (i) typically yields vacuous guarantees for overparameterized models. Furthermore, (ii) shrinks the training set and its guarantee erodes with each re-use of the holdout set. In this paper, we introduce a method that leverages unlabeled data to produce generalization bounds. After augmenting our (labeled) training set with randomly labeled fresh examples, we train in the standard fashion. Whenever classifiers achieve low error on clean data and high error on noisy data, our bound provides a tight upper bound on the true risk. We prove that our bound is valid for 0-1 empirical risk minimization and with linear classifiers trained by gradient descent. Our approach is especially useful in conjunction with deep learning due to the early learning phenomenon whereby networks fit true labels before noisy labels but requires one intuitive assumption. Empirically, on canonical computer vision and NLP tasks, our bound provides non-vacuous generalization guarantees that track actual performance closely. This work provides practitioners with an option for certifying the generalization of deep nets even when unseen labeled data is unavailable and provides theoretical insights into the relationship between random label noise and generalization.