mlsanityData to 7 Oct 2026

Predicting Out-of-Distribution Error with the Projection Norm

arXiv 2202.05834 ↗First posted 11 Feb 2022cs.LG stat.MLcited by 50

Not on the map. Why?

Abstract (arXiv)

We propose a metric -- Projection Norm -- to predict a model's performance on out-of-distribution (OOD) data without access to ground truth labels. Projection Norm first uses model predictions to pseudo-label test samples and then trains a new model on the pseudo-labels. The more the new model's parameters differ from an in-distribution model, the greater the predicted OOD error. Empirically, our approach outperforms existing methods on both image and text classification tasks and across different network architectures. Theoretically, we connect our approach to a bound on the test error for overparameterized linear models. Furthermore, we find that Projection Norm is the only approach that achieves non-trivial detection performance on adversarial examples. Our code is available at https://github.com/yaodongyu/ProjNorm.

Key facts. Cited by 50 papers in the map (data through 7 Oct 2026). Builds on 56 of its 76 references that are in the map. First posted on arXiv 11 Feb 2022.

Builds on

56 of 76 references.

References · 56

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  5. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  6. 2017On Calibration of Modern Neural Networkscited by 4,860
  7. 2015Domain-Adversarial Training of Neural Networkscited by 3,023
  8. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,857
  9. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  10. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  11. 2016A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networkscited by 2,362
  12. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,842
38 more
  1. 2015A large annotated corpus for learning natural language inferencecited by 1,837
  2. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,507
  3. 2018A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attackscited by 1,472
  4. 2018ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustnesscited by 1,430
  5. 2019Natural Adversarial Examplescited by 1,401
  6. 2020WILDS: A Benchmark of in-the-Wild Distribution Shiftscited by 1,366
  7. 2017Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networkscited by 1,325
  8. 2015Learning Transferable Features with Deep Adaptation Networkscited by 1,323
  9. 2017Adversarial Discriminative Domain Adaptationcited by 1,309
  10. 2016Adversarial Machine Learning at Scalecited by 934
  11. 2019Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inferencecited by 864
  12. 2017Selective Classification for Deep Neural Networkscited by 799
  13. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 732
  14. 2018Gradient Descent Finds Global Minima of Deep Neural Networkscited by 572
  15. 2019Benign Overfitting in Linear Regressioncited by 544
  16. 2017Adversarial Examples Are Not Easily Detected: Bypassing Ten Detection Methodscited by 542
  17. 2019Surprises in High-Dimensional Ridgeless Least Squares Interpolationcited by 508
  18. 2020Measuring Robustness to Natural Distribution Shifts in Image Classificationcited by 452
  19. 2018Learning and Generalization in Overparameterized Neural Networks, Going Beyond Two Layerscited by 340
  20. 2020Pretrained Transformers Improve Out-of-Distribution Robustnesscited by 293
  21. 2017The Marginal Value of Adaptive Gradient Methods in Machine Learningcited by 248
  22. 2018Stress Test Evaluation for Natural Language Inferencecited by 211
  23. 2018To Trust Or Not To Trust A Classifiercited by 205
  24. 2020BREEDS: Benchmarks for Subpopulation Shiftcited by 155
  25. 2020Benign overfitting in ridge regressioncited by 152
  26. 2022Leveraging Unlabeled Data to Predict Out-of-Distribution Performancecited by 141
  27. 2018Breaking NLI Systems with Sentences that Require Simple Lexical Inferencescited by 134
  28. 2021Assessing Generalization of SGD via Disagreementcited by 116
  29. 2021Predicting with Confidence on Unseen Distributionscited by 116
  30. 2020The Effect of Natural Distribution Shift on Question Answering Modelscited by 104
  31. 2020Are Labels Always Necessary for Classifier Accuracy Evaluation?cited by 104
  32. 2021What Does Rotation Prediction Tell Us about Classifier Accuracy under Varying Testing Environments?cited by 63
  33. 2019Do Image Classifiers Generalize Across Time?cited by 59
  34. 2020Estimating Generalization under Distribution Shifts via Domain-Invariant Representationscited by 56
  35. 2021Mandoline: Model Evaluation under Distribution Shiftcited by 55
  36. 2021Uniform Convergence of Interpolators: Gaussian Width, Norm Bounds, and Benign Overfittingcited by 54
  37. 2021Detecting Errors and Estimating Accuracy on Unlabeled Data with Self-training Ensemblescited by 54
  38. 2020Distributional Generalization: A New Kind of Generalizationcited by 41

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 5; 2023: 11; 2024: 20; 2025: 8; 2026: 6.

50 papers.

Sort:

By area:

  1. 2023Large Language Model Routing with Benchmark Datasetscited by 166
  2. 2022Agreement-on-the-Line: Predicting the Performance of Neural Networks under Distribution Shiftcited by 94
  3. 2024Mission Critical -- Satellite Data is a Distinct Modality in Machine Learningcited by 76
  4. 2022On the Strong Correlation Between Model Invariance and Generalizationcited by 23
  5. 2023Confidence and Dispersity Speak: Characterising Prediction Matrix for Unsupervised Accuracy Estimationcited by 22
  6. 2023Characterizing Out-of-Distribution Error via Optimal Transportcited by 20
  7. 2024A Survey on Evaluation of Out-of-Distribution Generalizationcited by 20
  8. 2022A Learning Based Hypothesis Test for Harmful Covariate Shiftcited by 18
  9. 2023GNNEvaluator: Evaluating GNN Performance On Unseen Graphs Without Labelscited by 17
  10. 2024Energy-based Automated Model Evaluationcited by 16
  11. 2022Information-Theoretic Analysis of Unsupervised Domain Adaptationcited by 15
  12. 2023On the Importance of Feature Separability in Predicting Out-Of-Distribution Errorcited by 15
38 more
  1. 2023Predicting Out-of-Distribution Error with Confidence Optimal Transportcited by 13
  2. 2024A Framework for Efficient Model Evaluation through Stratification, Sampling, and Estimationcited by 13
  3. 2023Chasing Fairness Under Distribution Shift: A Model Weight Perturbation Approachcited by 12
  4. 2024Online GNN Evaluation Under Test-time Graph Distribution Shiftscited by 12
  5. 2022Understanding new tasks through the lens of training data via exponential tiltingcited by 11
  6. 2023Estimating Large Language Model Capabilities without Labeled Test Datacited by 9
  7. 2024MANO: Exploiting Matrix Norm for Unsupervised Accuracy Estimation Under Distribution Shiftscited by 8
  8. 2024Predicting the Performance of Foundation Models via Agreement-on-the-Linecited by 7
  9. 2024Beyond the Federation: Topology-aware Federated Learning for Generalization to Unseen Clientscited by 7
  10. 2023CAME: Contrastive Automated Model Evaluationcited by 6
  11. 2024Out-of-Distribution Data: An Acquaintance of Adversarial Examples -- A Surveycited by 6
  12. 2024What Does Softmax Probability Tell Us about Classifiers Ranking Across Diverse Test Conditions?cited by 6
  13. 2023K-means Clustering Based Feature Consistency Alignment for Label-free Model Evaluationcited by 4
  14. 2024Towards Unsupervised Model Selection for Domain Adaptive Object Detectioncited by 4
  15. 2025Error Slice Discovery via Manifold Compactnesscited by 4
  16. 2025Suitability Filter: A Statistical Framework for Classifier Evaluation in Real-World Deployment Settingscited by 4
  17. 2024Leveraging Gradients for Unsupervised Accuracy Estimation under Distribution Shiftcited by 3
  18. 2024Unveiling AI's Blind Spots: An Oracle for In-Domain, Out-of-Domain, and Adversarial Errorscited by 3
  19. 2025Benchmarking Transferability: A Framework for Fair and Robust Evaluationcited by 2
  20. 2023Predicting generalization performance with correctness discriminatorscited by 1
  21. 2024Selection, Ensemble, and Adaptation: Advancing Multi-Source-Free Domain Adaptation via Architecture Zoocited by 1
  22. 2024Label-Free Model Failure Detection for Lidar-based Point Cloud Segmentationcited by 1
  23. 2024Source-Free Domain-Invariant Performance Predictioncited by 1
  24. 2024Poor-Supervised Evaluation for SuperLLM via Mutual Consistencycited by 1
  25. 2024Learning to Rank Pre-trained Vision-Language Models for Downstream Taskscited by 1
  26. 2025Shapley-Guided Utility Learning for Effective Graph Inference Data Valuationcited by 1
  27. 2025Confidence and Dispersity as Signals: Unsupervised Model Evaluation and Rankingcited by 1
  28. 2025ODP-Bench: Benchmarking Out-of-Distribution Performance Predictioncited by 1
  29. 2026TopoGeoScore: A Self-Supervised Source-Only Geometric Framework for OOD Checkpoint Selectioncited by 1
  30. 2026Learning to Evaluate: Cost-Effective Model Evaluation on Unlabeled Data with Meta-Learningcited by 1
  31. 2026Bridging Domain Expertise and Generalization for Performance Estimationcited by 1
  32. 2024Assessing Model Generalization in Vicinitycited by 0
  33. 2024Calibration of Network Confidence for Unsupervised Domain Adaptation Using Estimated Accuracycited by 0
  34. 2025ALSA: Anchors in Logit Space for Out-of-Distribution Accuracy Estimationcited by 0
  35. 2025Using Synthetic Data to estimate the True Error is theoretically and practically doablecited by 0
  36. 2026Co-Adaptive Multi-Task LoRA: Transfer-Aware, Label-Free Control of Domain Participationcited by 0
  37. 2026Can Circuit Alignment Predict OOD Generalization?cited by 0
  38. 2026STAMP: Predicting Out-of-Distribution Generalization without Target Datacited by 0