mlsanityData to 14 Oct 2026

What Does Rotation Prediction Tell Us about Classifier Accuracy under Varying Testing Environments?

arXiv 2106.05961 ↗First posted 10 Jun 2021cs.CV cs.LGcited by 63

Not on the map. Why?

Abstract (arXiv)

Understanding classifier decision under novel environments is central to the community, and a common practice is evaluating it on labeled test sets. However, in real-world testing, image annotations are difficult and expensive to obtain, especially when the test environment is changing. A natural question then arises: given a trained classifier, can we evaluate its accuracy on varying unlabeled test sets? In this work, we train semantic classification and rotation prediction in a multi-task way. On a series of datasets, we report an interesting finding, i.e., the semantic classification accuracy exhibits a strong linear relationship with the accuracy of the rotation prediction task (Pearson's Correlation r > 0.88). This finding allows us to utilize linear regression to estimate classifier performance from the accuracy of rotation prediction which can be obtained on the test set through the freely generated rotation labels.

Key facts. Cited by 63 papers in the map (data through 14 Oct 2026). Builds on 38 of its 57 references that are in the map. First posted on arXiv 10 Jun 2021.

Builds on

38 of 57 references.

References · 38

  1. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  2. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  3. 2016Densely Connected Convolutional Networkscited by 6,720
  4. 2017On Calibration of Modern Neural Networkscited by 4,960
  5. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,877
  6. 2016A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networkscited by 2,385
  7. 2017What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?cited by 2,352
  8. 2018Unsupervised Representation Learning by Predicting Image Rotationscited by 1,591
  9. 2018A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attackscited by 1,485
  10. 2016Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzlescited by 1,382
  11. 2017Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networkscited by 1,328
  12. 2019Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shiftcited by 1,287
26 more
  1. 2018Deep Clustering for Unsupervised Learning of Visual Featurescited by 1,028
  2. 2017Exploring Generalization in Deep Learningcited by 600
  3. 2019Using Self-Supervised Learning Can Improve Model Robustness and Uncertaintycited by 514
  4. 2019Fantastic Generalization Measures and Where to Find Themcited by 513
  5. 2020Measuring Robustness to Natural Distribution Shifts in Image Classificationcited by 454
  6. 2019Likelihood Ratios for Out-of-Distribution Detectioncited by 453
  7. 2019Domain Generalization by Solving Jigsaw Puzzlescited by 450
  8. 2017VisDA: The Visual Domain Adaptation Challengecited by 435
  9. 2020Generalized ODIN: Detecting Out-of-distribution Image without Learning from Out-of-distribution Datacited by 408
  10. 2019S4L: Self-Supervised Semi-Supervised Learningcited by 335
  11. 2018Stronger generalization bounds for deep nets via a compression approachcited by 307
  12. 2019Revisiting Self-Supervised Visual Representation Learningcited by 251
  13. 2017Multi-task Self-Supervised Visual Learningcited by 200
  14. 2018Do CIFAR-10 Classifiers Generalize to CIFAR-10?cited by 200
  15. 2017Colorization as a Proxy Task for Visual Understandingcited by 175
  16. 2018Making Sense of Vision and Touch: Self-Supervised Learning of Multimodal Representations for Contact-Rich Taskscited by 154
  17. 2019Unsupervised Domain Adaptation through Self-Supervisioncited by 116
  18. 2018Predicting the Generalization Gap in Deep Networks with Margin Distributionscited by 113
  19. 2020Predicting Neural Network Accuracy from Weightscited by 110
  20. 2020Are Labels Always Necessary for Classifier Accuracy Evaluation?cited by 104
  21. 2017Unsupervised Learning by Predicting Noisecited by 98
  22. 2020Classifying the classifier: dissecting the weight space of neural networkscited by 65
  23. 2017Cross-Domain Self-supervised Multi-task Feature Learning using Synthetic Imagerycited by 56
  24. 2020Computing the Testing Error without a Testing Setcited by 54
  25. 2020Self-Supervised Learning Across Domainscited by 44
  26. 2017Estimating Accuracy from Unlabeled Data: A Probabilistic Logic Approachcited by 27

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 4; 2022: 15; 2023: 19; 2024: 16; 2025: 7; 2026: 2.

63 papers.

Sort:

By area:

  1. 2022Leveraging Unlabeled Data to Predict Out-of-Distribution Performancecited by 145
  2. 2021Predicting with Confidence on Unseen Distributionscited by 116
  3. 2022Agreement-on-the-Line: Predicting the Performance of Neural Networks under Distribution Shiftcited by 94
  4. 2021Detecting Errors and Estimating Accuracy on Unlabeled Data with Self-training Ensemblescited by 54
  5. 2022Predicting Out-of-Distribution Error with the Projection Normcited by 50
  6. 2022Active Surrogate Estimators: An Active Learning Approach to Label-Efficient Model Evaluationcited by 26
  7. 2022Predicting is not Understanding: Recognizing and Addressing Underspecification in Machine Learningcited by 26
  8. 2022On the Strong Correlation Between Model Invariance and Generalizationcited by 23
  9. 2023Confidence and Dispersity Speak: Characterising Prediction Matrix for Unsupervised Accuracy Estimationcited by 22
  10. 2024A Survey on Evaluation of Out-of-Distribution Generalizationcited by 21
  11. 2023Characterizing Out-of-Distribution Error via Optimal Transportcited by 20
  12. 2022Estimating and Explaining Model Performance When Both Covariates and Labels Shiftcited by 19
38 more
  1. 2023GNNEvaluator: Evaluating GNN Performance On Unseen Graphs Without Labelscited by 17
  2. 2024Energy-based Automated Model Evaluationcited by 17
  3. 2023(Almost) Provable Error Bounds Under Distribution Shift via Disagreement Discrepancycited by 16
  4. 2023On the Importance of Feature Separability in Predicting Out-Of-Distribution Errorcited by 15
  5. 2024A Framework for Efficient Model Evaluation through Stratification, Sampling, and Estimationcited by 14
  6. 2024Online GNN Evaluation Under Test-time Graph Distribution Shiftscited by 12
  7. 2023A Bag-of-Prototypes Representation for Dataset-Level Applicationscited by 11
  8. 2024Self-supervised visual learning in the low-data regime: a comparative evaluationcited by 11
  9. 2021Label-Free Model Evaluation with Semi-Structured Dataset Representationscited by 10
  10. 2022Attribute Descent: Simulating Object-Centric Datasets on the Content Level and Beyondcited by 9
  11. 2021Ranking Models in Unlabeled New Environmentscited by 8
  12. 2024MANO: Exploiting Matrix Norm for Unsupervised Accuracy Estimation Under Distribution Shiftscited by 8
  13. 2023ProbVLM: Probabilistic Adapter for Frozen Vision-Language Modelscited by 7
  14. 2023Selecting which Dense Retriever to use for Zero-Shot Searchcited by 7
  15. 2024Predicting the Performance of Foundation Models via Agreement-on-the-Linecited by 7
  16. 2022HAPI: A Large-scale Longitudinal Dataset of Commercial ML API Predictionscited by 6
  17. 2023CAME: Contrastive Automated Model Evaluationcited by 6
  18. 2024Bounding Box Stability against Feature Dropout Reflects Detector Generalization across Environmentscited by 6
  19. 2022What Images are More Memorable to Machines?cited by 5
  20. 2022Predicting Out-of-Domain Generalization with Neighborhood Invariancecited by 4
  21. 2022Composite Learning for Robust and Effective Dense Predictionscited by 4
  22. 2023Trust, but Verify: Using Self-Supervised Probing to Improve Trustworthinesscited by 4
  23. 2023Unsupervised Evaluation of Out-of-distribution Detection: A Data-centric Perspectivecited by 4
  24. 2023Toward Auto-evaluation with Confidence-based Category Relation-aware Regressioncited by 4
  25. 2023K-means Clustering Based Feature Consistency Alignment for Label-free Model Evaluationcited by 4
  26. 2025Error Slice Discovery via Manifold Compactnesscited by 4
  27. 2025Suitability Filter: A Statistical Framework for Classifier Evaluation in Real-World Deployment Settingscited by 4
  28. 2022Performance Prediction Under Dataset Shiftcited by 3
  29. 2022Test-time Recalibration of Conformal Predictors Under Distribution Shift Based on Unlabeled Examplescited by 3
  30. 2023Unsupervised Accuracy Estimation of Deep Visual Models using Domain-Adaptive Adversarial Perturbation without Source Samplescited by 3
  31. 2024Leveraging Gradients for Unsupervised Accuracy Estimation under Distribution Shiftcited by 3
  32. 2024Unveiling AI's Blind Spots: An Oracle for In-Domain, Out-of-Domain, and Adversarial Errorscited by 3
  33. 2025Self-Supervised Learning for Image Segmentation: A Comprehensive Surveycited by 3
  34. 2023Distance Matters For Improving Performance Estimation Under Covariate Shiftcited by 2
  35. 2023Heterogeneous Generative Knowledge Distillation with Masked Image Modelingcited by 2
  36. 2024Temporal Generalization Estimation in Evolving Graphscited by 2
  37. 2022Improving Self-supervised Learning for Out-of-distribution Task via Auxiliary Classifiercited by 1
  38. 2023On the Transferability of Learning Models for Semantic Segmentation for Remote Sensing Datacited by 1

50 most cited of 63 All 63 (JSON)