# Label-Free Model Evaluation with Semi-Structured Dataset Representations

arXiv: https://arxiv.org/abs/2112.00694 · HTML: https://mlsanity.com/paper/2112.00694/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 46 references that are in the map.
- First posted on arXiv 1 Dec 2021.

## Builds on (most cited first, 30 of 30)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [Benchmarking Neural Network Robustness to Common Corruptions and Perturbations](https://mlsanity.com/paper/1903.12261/) (2019)
- [A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks](https://mlsanity.com/paper/1610.02136/) (2016)
- [Unsupervised Representation Learning by Predicting Image Rotations](https://mlsanity.com/paper/1803.07728/) (2018)
- [Deep Sets](https://mlsanity.com/paper/1703.06114/) (2017)
- [Moment Matching for Multi-Source Domain Adaptation](https://mlsanity.com/paper/1812.01754/) (2018)
- [Energy-based Out-of-distribution Detection](https://mlsanity.com/paper/2010.03759/) (2020)
- [Network Dissection: Quantifying Interpretability of Deep Visual Representations](https://mlsanity.com/paper/1704.05796/) (2017)
- [Fantastic Generalization Measures and Where to Find Them](https://mlsanity.com/paper/1912.02178/) (2019)
- [Training Confidence-calibrated Classifiers for Detecting Out-of-Distribution Samples](https://mlsanity.com/paper/1711.09325/) (2017)
- [Semi-supervised Domain Adaptation via Minimax Entropy](https://mlsanity.com/paper/1904.06487/) (2019)
- [Learning Confidence for Out-of-Distribution Detection in Neural Networks](https://mlsanity.com/paper/1802.04865/) (2018)
- [SO-Net: Self-Organizing Network for Point Cloud Analysis](https://mlsanity.com/paper/1803.04249/) (2018)
- [Deep Stable Learning for Out-Of-Distribution Generalization](https://mlsanity.com/paper/2104.07876/) (2021)
- [Predicting with Confidence on Unseen Distributions](https://mlsanity.com/paper/2107.03315/) (2021)
- [Predicting the Generalization Gap in Deep Networks with Margin Distributions](https://mlsanity.com/paper/1810.00113/) (2018)
- [Are Labels Always Necessary for Classifier Accuracy Evaluation?](https://mlsanity.com/paper/2007.02915/) (2020)
- [MOOD: Multi-level Out-of-distribution Detection](https://mlsanity.com/paper/2104.14726/) (2021)
- [In Search of Robust Measures of Generalization](https://mlsanity.com/paper/2010.11924/) (2020)
- [What Does Rotation Prediction Tell Us about Classifier Accuracy under Varying Testing Environments?](https://mlsanity.com/paper/2106.05961/) (2021)
- [Computing the Testing Error without a Testing Set](https://mlsanity.com/paper/2005.00450/) (2020)
- [Detecting Errors and Estimating Accuracy on Unlabeled Data with Self-training Ensembles](https://mlsanity.com/paper/2106.15728/) (2021)
- [Estimating Accuracy from Unlabeled Data: A Probabilistic Logic Approach](https://mlsanity.com/paper/1705.07086/) (2017)
- [Neural Data Server: A Large-Scale Search Engine for Transfer Learning Data](https://mlsanity.com/paper/2001.02799/) (2020)
- [Rep the Set: Neural Networks for Learning Set Representations](https://mlsanity.com/paper/1904.01962/) (2019)
- [Distribution-Free Distribution Regression](https://mlsanity.com/paper/1302.0082/) (2013)
- [Unsupervised Risk Estimation Using Only Conditional Independence Structure](https://mlsanity.com/paper/1606.05313/) (2016)
- [Set Representation Learning with Generalized Sliced-Wasserstein Embeddings](https://mlsanity.com/paper/2103.03892/) (2021)

## Built on by (most cited first, 10 of 10)

- [Active Surrogate Estimators: An Active Learning Approach to Label-Efficient Model Evaluation](https://mlsanity.com/paper/2202.06881/) (2022)
- [Energy-based Automated Model Evaluation](https://mlsanity.com/paper/2401.12689/) (2024)
- [Transfer Knowledge from Head to Tail: Uncertainty Calibration under Long-tailed Distribution](https://mlsanity.com/paper/2304.06537/) (2023)
- [CIFAR-10-Warehouse: Broad and More Realistic Testbeds in Model Generalization Analysis](https://mlsanity.com/paper/2310.04414/) (2023)
- [CAME: Contrastive Automated Model Evaluation](https://mlsanity.com/paper/2308.11111/) (2023)
- [Bounding Box Stability against Feature Dropout Reflects Detector Generalization across Environments](https://mlsanity.com/paper/2403.13803/) (2024)
- [Unsupervised Evaluation of Out-of-distribution Detection: A Data-centric Perspective](https://mlsanity.com/paper/2302.08287/) (2023)
- [Toward Auto-evaluation with Confidence-based Category Relation-aware Regression](https://mlsanity.com/paper/2304.08288/) (2023)
- [Suitability Filter: A Statistical Framework for Classifier Evaluation in Real-World Deployment Settings](https://mlsanity.com/paper/2505.22356/) (2025)
- [Domain-adaptive and Subgroup-specific Cascaded Temperature Regression for Out-of-distribution Calibration](https://mlsanity.com/paper/2402.09204/) (2024)

## Abstract (arXiv)

Label-free model evaluation, or AutoEval, estimates model accuracy on unlabeled test sets, and is critical for understanding model behaviors in various unseen environments. In the absence of image labels, based on dataset representations, we estimate model performance for AutoEval with regression. On the one hand, image feature is a straightforward choice for such representations, but it hampers regression learning due to being unstructured (ie no specific meanings for component at certain location) and of large-scale. On the other hand, previous methods adopt simple structured representations (like average confidence or average feature), but insufficient to capture the data characteristics given their limited dimensions. In this work, we take the best of both worlds and propose a new semi-structured dataset representation that is manageable for regression learning while containing rich information for AutoEval. Based on image features, we integrate distribution shapes, clusters, and representative samples for a semi-structured dataset representation. Besides the structured overall description with distribution shapes, the unstructured description with clusters and representative samples include additional fine-grained information facilitating the AutoEval task. On three existing datasets and 25 newly introduced ones, we experimentally show that the proposed representation achieves competitive results. Code and dataset are available at https://github.com/sxzrt/Semi-Structured-Dataset-Representations.