# Assessing Model Generalization in Vicinity

arXiv: https://arxiv.org/abs/2406.09257 · HTML: https://mlsanity.com/paper/2406.09257/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 29 of its 42 references that are in the map.
- First posted on arXiv 13 Jun 2024.

## Builds on (most cited first, 29 of 29)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features](https://mlsanity.com/paper/1905.04899/) (2019)
- [Benchmarking Neural Network Robustness to Common Corruptions and Perturbations](https://mlsanity.com/paper/1903.12261/) (2019)
- [A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks](https://mlsanity.com/paper/1610.02136/) (2016)
- [The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization](https://mlsanity.com/paper/2006.16241/) (2020)
- [Do ImageNet Classifiers Generalize to ImageNet?](https://mlsanity.com/paper/1902.10811/) (2019)
- [Natural Adversarial Examples](https://mlsanity.com/paper/1907.07174/) (2019)
- [Learning Robust Global Representations by Penalizing Local Predictive Power](https://mlsanity.com/paper/1905.13549/) (2019)
- [Puzzle Mix: Exploiting Saliency and Local Statistics for Optimal Mixup](https://mlsanity.com/paper/2009.06962/) (2020)
- [Do CIFAR-10 Classifiers Generalize to CIFAR-10?](https://mlsanity.com/paper/1806.00451/) (2018)
- [Leveraging Unlabeled Data to Predict Out-of-Distribution Performance](https://mlsanity.com/paper/2201.04234/) (2022)
- [Assessing Generalization of SGD via Disagreement](https://mlsanity.com/paper/2106.13799/) (2021)
- [Predicting with Confidence on Unseen Distributions](https://mlsanity.com/paper/2107.03315/) (2021)
- [Are Labels Always Necessary for Classifier Accuracy Evaluation?](https://mlsanity.com/paper/2007.02915/) (2020)
- [The iWildCam 2020 Competition Dataset](https://mlsanity.com/paper/2004.10340/) (2020)
- [ResizeMix: Mixing Data with Preserved Object Information and True Labels](https://mlsanity.com/paper/2012.11101/) (2020)
- [What Does Rotation Prediction Tell Us about Classifier Accuracy under Varying Testing Environments?](https://mlsanity.com/paper/2106.05961/) (2021)
- [Predicting Out-of-Distribution Error with the Projection Norm](https://mlsanity.com/paper/2202.05834/) (2022)
- [In Defense of Uniform Convergence: Generalization via derandomization with an application to interpolating predictors](https://mlsanity.com/paper/1912.04265/) (2019)
- [On Uniform Convergence and Low-Norm Interpolation Learning](https://mlsanity.com/paper/2006.05942/) (2020)
- [On the Strong Correlation Between Model Invariance and Generalization](https://mlsanity.com/paper/2207.07065/) (2022)
- [RATT: Leveraging Unlabeled Data to Guarantee Generalization](https://mlsanity.com/paper/2105.00303/) (2021)
- [Characterizing Out-of-Distribution Error via Optimal Transport](https://mlsanity.com/paper/2305.15640/) (2023)
- [On the Importance of Feature Separability in Predicting Out-Of-Distribution Error](https://mlsanity.com/paper/2303.15488/) (2023)
- [Robustness to Augmentations as a Generalization metric](https://mlsanity.com/paper/2101.06459/) (2021)
- [Revisiting Vicinal Risk Minimization for Partially Supervised Multi-Label Classification Under Data Scarcity](https://mlsanity.com/paper/2204.08954/) (2022)
- [Generalization Bounds for Vicinal Risk Minimization Principle](https://mlsanity.com/paper/1811.04351/) (2018)

## Abstract (arXiv)

This paper evaluates the generalization ability of classification models on out-of-distribution test sets without depending on ground truth labels. Common approaches often calculate an unsupervised metric related to a specific model property, like confidence or invariance, which correlates with out-of-distribution accuracy. However, these metrics are typically computed for each test sample individually, leading to potential issues caused by spurious model responses, such as overly high or low confidence. To tackle this challenge, we propose incorporating responses from neighboring test samples into the correctness assessment of each individual sample. In essence, if a model consistently demonstrates high correctness scores for nearby samples, it increases the likelihood of correctly predicting the target sample, and vice versa. The resulting scores are then averaged across all test samples to provide a holistic indication of model accuracy. Developed under the vicinal risk formulation, this approach, named vicinal risk proxy (VRP), computes accuracy without relying on labels. We show that applying the VRP method to existing generalization indicators, such as average confidence and effective invariance, consistently improves over these baselines both methodologically and experimentally. This yields a stronger correlation with model accuracy, especially on challenging out-of-distribution test sets.