mlsanityData to 14 Oct 2026

Label-Free Model Evaluation with Semi-Structured Dataset Representations

arXiv 2112.00694 ↗First posted 1 Dec 2021cs.CVcited by 10

Not on the map. Why?

Abstract (arXiv)

Label-free model evaluation, or AutoEval, estimates model accuracy on unlabeled test sets, and is critical for understanding model behaviors in various unseen environments. In the absence of image labels, based on dataset representations, we estimate model performance for AutoEval with regression. On the one hand, image feature is a straightforward choice for such representations, but it hampers regression learning due to being unstructured (ie no specific meanings for component at certain location) and of large-scale. On the other hand, previous methods adopt simple structured representations (like average confidence or average feature), but insufficient to capture the data characteristics given their limited dimensions. In this work, we take the best of both worlds and propose a new semi-structured dataset representation that is manageable for regression learning while containing rich information for AutoEval. Based on image features, we integrate distribution shapes, clusters, and representative samples for a semi-structured dataset representation. Besides the structured overall description with distribution shapes, the unstructured description with clusters and representative samples include additional fine-grained information facilitating the AutoEval task. On three existing datasets and 25 newly introduced ones, we experimentally show that the proposed representation achieves competitive results. Code and dataset are available at https://github.com/sxzrt/Semi-Structured-Dataset-Representations.

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Builds on 30 of its 46 references that are in the map. First posted on arXiv 1 Dec 2021.

Builds on

30 of 46 references.

References · 30

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  3. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,248
  4. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,877
  5. 2016A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networkscited by 2,385
  6. 2018Unsupervised Representation Learning by Predicting Image Rotationscited by 1,591
  7. 2017Deep Setscited by 1,421
  8. 2018Moment Matching for Multi-Source Domain Adaptationcited by 1,331
  9. 2020Energy-based Out-of-distribution Detectioncited by 1,299
  10. 2017Network Dissection: Quantifying Interpretability of Deep Visual Representationscited by 708
  11. 2019Fantastic Generalization Measures and Where to Find Themcited by 513
  12. 2017Training Confidence-calibrated Classifiers for Detecting Out-of-Distribution Samplescited by 389
18 more
  1. 2019Semi-supervised Domain Adaptation via Minimax Entropycited by 335
  2. 2018Learning Confidence for Out-of-Distribution Detection in Neural Networkscited by 245
  3. 2018SO-Net: Self-Organizing Network for Point Cloud Analysiscited by 210
  4. 2021Deep Stable Learning for Out-Of-Distribution Generalizationcited by 153
  5. 2021Predicting with Confidence on Unseen Distributionscited by 116
  6. 2018Predicting the Generalization Gap in Deep Networks with Margin Distributionscited by 113
  7. 2020Are Labels Always Necessary for Classifier Accuracy Evaluation?cited by 104
  8. 2021MOOD: Multi-level Out-of-distribution Detectioncited by 94
  9. 2020In Search of Robust Measures of Generalizationcited by 79
  10. 2021What Does Rotation Prediction Tell Us about Classifier Accuracy under Varying Testing Environments?cited by 63
  11. 2020Computing the Testing Error without a Testing Setcited by 54
  12. 2021Detecting Errors and Estimating Accuracy on Unlabeled Data with Self-training Ensemblescited by 54
  13. 2017Estimating Accuracy from Unlabeled Data: A Probabilistic Logic Approachcited by 27
  14. 2020Neural Data Server: A Large-Scale Search Engine for Transfer Learning Datacited by 27
  15. 2019Rep the Set: Neural Networks for Learning Set Representationscited by 22
  16. 2013Distribution-Free Distribution Regressioncited by 21
  17. 2016Unsupervised Risk Estimation Using Only Conditional Independence Structurecited by 17
  18. 2021Set Representation Learning with Generalized Sliced-Wasserstein Embeddingscited by 4

Built on by

Cited by year

Citing papers per year: 2022: 1; 2023: 5; 2024: 3; 2025: 1.

10 papers.

  1. 2022Active Surrogate Estimators: An Active Learning Approach to Label-Efficient Model Evaluationcited by 26
  2. 2024Energy-based Automated Model Evaluationcited by 17
  3. 2023Transfer Knowledge from Head to Tail: Uncertainty Calibration under Long-tailed Distributioncited by 12
  4. 2023CIFAR-10-Warehouse: Broad and More Realistic Testbeds in Model Generalization Analysiscited by 12
  5. 2023CAME: Contrastive Automated Model Evaluationcited by 6
  6. 2024Bounding Box Stability against Feature Dropout Reflects Detector Generalization across Environmentscited by 6
  7. 2023Unsupervised Evaluation of Out-of-distribution Detection: A Data-centric Perspectivecited by 4
  8. 2023Toward Auto-evaluation with Confidence-based Category Relation-aware Regressioncited by 4
  9. 2025Suitability Filter: A Statistical Framework for Classifier Evaluation in Real-World Deployment Settingscited by 4
  10. 2024Domain-adaptive and Subgroup-specific Cascaded Temperature Regression for Out-of-distribution Calibrationcited by 1