# Ranking Models in Unlabeled New Environments

arXiv: https://arxiv.org/abs/2108.10310 · HTML: https://mlsanity.com/paper/2108.10310/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 63 references that are in the map.
- First posted on arXiv 23 Aug 2021.

## Builds on (most cited first, 30 of 35)

- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [Domain-Adversarial Training of Neural Networks](https://mlsanity.com/paper/1505.07818/) (2015)
- [Learning Transferable Features with Deep Adaptation Networks](https://mlsanity.com/paper/1502.02791/) (2015)
- [CyCADA: Cycle-Consistent Adversarial Domain Adaptation](https://mlsanity.com/paper/1711.03213/) (2017)
- [StarGAN: Unified Generative Adversarial Networks for Multi-Domain Image-to-Image Translation](https://mlsanity.com/paper/1711.09020/) (2017)
- [Performance Measures and a Data Set for Multi-Target, Multi-Camera Tracking](https://mlsanity.com/paper/1609.01775/) (2016)
- [Exploring Generalization in Deep Learning](https://mlsanity.com/paper/1706.08947/) (2017)
- [Do Better ImageNet Models Transfer Better?](https://mlsanity.com/paper/1805.08974/) (2018)
- [Person Transfer GAN to Bridge Domain Gap for Person Re-Identification](https://mlsanity.com/paper/1711.08565/) (2017)
- [Beyond Part Models: Person Retrieval with Refined Part Pooling (and a Strong Convolutional Baseline)](https://mlsanity.com/paper/1711.09349/) (2017)
- [Stronger generalization bounds for deep nets via a compression approach](https://mlsanity.com/paper/1802.05296/) (2018)
- [Unlabeled Samples Generated by GAN Improve the Person Re-identification Baseline in vitro](https://mlsanity.com/paper/1701.07717/) (2017)
- [WebVision Database: Visual Learning and Understanding from Web Data](https://mlsanity.com/paper/1708.02862/) (2017)
- [Person Re-identification: Past, Present and Future](https://mlsanity.com/paper/1610.02984/) (2016)
- [Unbiased Learning-to-Rank with Biased Feedback](https://mlsanity.com/paper/1608.04468/) (2016)
- [Invariance Matters: Exemplar Memory for Domain Adaptive Person Re-identification](https://mlsanity.com/paper/1904.01990/) (2019)
- [Meta-Sim: Learning to Generate Synthetic Datasets](https://mlsanity.com/paper/1904.11621/) (2019)
- [Predicting the Generalization Gap in Deep Networks with Margin Distributions](https://mlsanity.com/paper/1810.00113/) (2018)
- [Deep Metric Learning with Hierarchical Triplet Loss](https://mlsanity.com/paper/1810.06951/) (2018)
- [Are Labels Always Necessary for Classifier Accuracy Evaluation?](https://mlsanity.com/paper/2007.02915/) (2020)
- [Dissecting Person Re-identification from the Viewpoint of Viewpoint](https://mlsanity.com/paper/1812.02162/) (2018)
- [Unsupervised Person Re-identification: Clustering and Fine-tuning](https://mlsanity.com/paper/1705.10444/) (2017)
- [Maximum-Entropy Fine-Grained Classification](https://mlsanity.com/paper/1809.05934/) (2018)
- [What Does Rotation Prediction Tell Us about Classifier Accuracy under Varying Testing Environments?](https://mlsanity.com/paper/2106.05961/) (2021)
- [Computing the Testing Error without a Testing Set](https://mlsanity.com/paper/2005.00450/) (2020)
- [Learning to Adapt Invariance in Memory for Person Re-identification](https://mlsanity.com/paper/1908.00485/) (2019)
- [Surpassing Real-World Source Training Data: Random 3D Characters for Generalizable Person Re-Identification](https://mlsanity.com/paper/2006.12774/) (2020)
- [Simulating Content Consistent Vehicle Datasets with Attribute Descent](https://mlsanity.com/paper/1912.08855/) (2019)
- [UnrealPerson: An Adaptive Pipeline towards Costless Person Re-identification](https://mlsanity.com/paper/2012.04268/) (2020)
- [Estimating Accuracy from Unlabeled Data: A Probabilistic Logic Approach](https://mlsanity.com/paper/1705.07086/) (2017)

## Built on by (most cited first, 8 of 8)

- [LOVM: Language-Only Vision Model Selection](https://mlsanity.com/paper/2306.08893/) (2023)
- [Unveiling Code Pre-Trained Models: Investigating Syntax and Semantics Capacities](https://mlsanity.com/paper/2212.10017/) (2022)
- [Unsupervised Evaluation of Out-of-distribution Detection: A Data-centric Perspective](https://mlsanity.com/paper/2302.08287/) (2023)
- [Alice Benchmarks: Connecting Real World Re-Identification with the Synthetic](https://mlsanity.com/paper/2310.04416/) (2023)
- [Large-scale Training Data Search for Object Re-identification](https://mlsanity.com/paper/2303.16186/) (2023)
- [PAS: Estimating the target accuracy before domain adaptation](https://mlsanity.com/paper/2604.09863/) (2026)
- [Model Evaluation in the Dark: Robust Classifier Metrics with Missing Labels](https://mlsanity.com/paper/2504.18385/) (2025)
- [Model Specific Task Similarity for Vision Language Model Selection via Layer Conductance](https://mlsanity.com/paper/2602.01346/) (2026)

## Abstract (arXiv)

Consider a scenario where we are supplied with a number of ready-to-use models trained on a certain source domain and hope to directly apply the most appropriate ones to different target domains based on the models' relative performance. Ideally we should annotate a validation set for model performance assessment on each new target environment, but such annotations are often very expensive. Under this circumstance, we introduce the problem of ranking models in unlabeled new environments. For this problem, we propose to adopt a proxy dataset that 1) is fully labeled and 2) well reflects the true model rankings in a given target environment, and use the performance rankings on the proxy sets as surrogates. We first select labeled datasets as the proxy. Specifically, datasets that are more similar to the unlabeled target domain are found to better preserve the relative performance rankings. Motivated by this, we further propose to search the proxy set by sampling images from various datasets that have similar distributions as the target. We analyze the problem and its solutions on the person re-identification (re-ID) task, for which sufficient datasets are publicly available, and show that a carefully constructed proxy set effectively captures relative performance ranking in new environments. Code is available at urlhttps://github.com/sxzrt/Proxy-Set.