mlsanityData to 14 Oct 2026

CAME: Contrastive Automated Model Evaluation

arXiv 2308.11111 ↗First posted 22 Aug 2023cs.CV cs.AI cs.LGcited by 6

Not on the map. Why?

Abstract (arXiv)

The Automated Model Evaluation (AutoEval) framework entertains the possibility of evaluating a trained machine learning model without resorting to a labeled testing set. Despite the promise and some decent results, the existing AutoEval methods heavily rely on computing distribution shifts between the unlabelled testing set and the training set. We believe this reliance on the training set becomes another obstacle in shipping this technology to real-world ML development. In this work, we propose Contrastive Automatic Model Evaluation (CAME), a novel AutoEval framework that is rid of involving training set in the loop. The core idea of CAME bases on a theoretical analysis which bonds the model performance with a contrastive loss. Further, with extensive empirical validation, we manage to set up a predictable relationship between the two, simply by deducing on the unlabeled/unseen testing set. The resulting framework CAME establishes a new SOTA results for AutoEval by surpassing prior work significantly.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 47 of its 61 references that are in the map. First posted on arXiv 22 Aug 2023.

Builds on

47 of 61 references.

References · 47

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  3. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  4. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  5. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  6. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  7. 2016Densely Connected Convolutional Networkscited by 6,720
  8. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  9. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,926
  10. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,877
  11. 2020Exploring Simple Siamese Representation Learningcited by 2,866
  12. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,857
35 more
  1. 2016A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networkscited by 2,385
  2. 2017What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?cited by 2,352
  3. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,287
  4. 2017Improved Regularization of Convolutional Neural Networks with Cutoutcited by 1,628
  5. 2018A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attackscited by 1,485
  6. 2016Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzlescited by 1,382
  7. 2015Learning Transferable Features with Deep Adaptation Networkscited by 1,329
  8. 2017Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networkscited by 1,328
  9. 2020Energy-based Out-of-distribution Detectioncited by 1,299
  10. 2019A Theoretical Analysis of Contrastive Unsupervised Representation Learningcited by 530
  11. 2019Scaling Out-of-Distribution Detection for Real-World Settingscited by 465
  12. 2017VisDA: The Visual Domain Adaptation Challengecited by 435
  13. 2020Generalized ODIN: Detecting Out-of-distribution Image without Learning from Out-of-distribution Datacited by 408
  14. 2015Return of Frustratingly Easy Domain Adaptationcited by 391
  15. 2018Do CIFAR-10 Classifiers Generalize to CIFAR-10?cited by 200
  16. 2020Predicting What You Already Know Helps: Provable Self-Supervised Learningcited by 152
  17. 2021Model Adaptation: Historical Contrastive Learning for Unsupervised Domain Adaptation without Source Datacited by 152
  18. 2022Leveraging Unlabeled Data to Predict Out-of-Distribution Performancecited by 145
  19. 2021Assessing Generalization of SGD via Disagreementcited by 117
  20. 2021Predicting with Confidence on Unseen Distributionscited by 116
  21. 2018Predicting the Generalization Gap in Deep Networks with Margin Distributionscited by 113
  22. 2020Predicting Neural Network Accuracy from Weightscited by 110
  23. 2022Chaos is a Ladder: A New Theoretical Understanding of Contrastive Learning via Augmentation Overlapcited by 105
  24. 2020Are Labels Always Necessary for Classifier Accuracy Evaluation?cited by 104
  25. 2021What Does Rotation Prediction Tell Us about Classifier Accuracy under Varying Testing Environments?cited by 63
  26. 2021A Prototype-Oriented Framework for Unsupervised Domain Adaptationcited by 62
  27. 2020Estimating Generalization under Distribution Shifts via Domain-Invariant Representationscited by 56
  28. 2021Mandoline: Model Evaluation under Distribution Shiftcited by 55
  29. 2020Computing the Testing Error without a Testing Setcited by 54
  30. 2021Detecting Errors and Estimating Accuracy on Unlabeled Data with Self-training Ensemblescited by 54
  31. 2022Predicting Out-of-Distribution Error with the Projection Normcited by 50
  32. 2021On the Surrogate Gap between Contrastive and Supervised Lossescited by 25
  33. 2021Label-Free Model Evaluation with Semi-Structured Dataset Representationscited by 10
  34. 2021On Predicting Generalization using GANscited by 6
  35. 2019Abstraction Mechanisms Predict Generalization in Deep Neural Networkscited by 5

Built on by

Cited by year

Citing papers per year: 2024: 4; 2025: 2.

6 papers.

  1. 2024Energy-based Automated Model Evaluationcited by 17
  2. 2024Estimating Model Performance Under Covariate Shift Without Labelscited by 9
  3. 2024Towards Unsupervised Model Selection for Domain Adaptive Object Detectioncited by 4
  4. 2024Pre-Trained Model Recommendation for Downstream Fine-tuningcited by 1
  5. 2025Automated Model Evaluation for Object Detection via Prediction Consistency and Reliabilitycited by 0
  6. 2025Using Synthetic Data to estimate the True Error is theoretically and practically doablecited by 0