mlsanityData to 14 Oct 2026

All models are wrong, some are useful: Model Selection with Limited Labels

arXiv 2410.13609 ↗First posted 17 Oct 2024cs.LGcited by 8

Not on the map. Why?

Abstract (arXiv)

We introduce MODEL SELECTOR, a framework for label-efficient selection of pretrained classifiers. Given a pool of unlabeled target data, MODEL SELECTOR samples a small subset of highly informative examples for labeling, in order to efficiently identify the best pretrained model for deployment on this target dataset. Through extensive experiments, we demonstrate that MODEL SELECTOR drastically reduces the need for labeled data while consistently picking the best or near-best performing model. Across 18 model collections on 16 different datasets, comprising over 1,500 pretrained models, MODEL SELECTOR reduces the labeling cost by up to 94.15% to identify the best model compared to the cost of the strongest baseline. Our results further highlight the robustness of MODEL SELECTOR in model selection, as it reduces the labeling cost by up to 72.41% when selecting a near-best model, whose accuracy is only within 1% of the best model.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Builds on 37 of its 87 references that are in the map. First posted on arXiv 17 Oct 2024.

Builds on

37 of 87 references.

References · 37

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2020Language Models are Few-Shot Learnerscited by 32,419
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  6. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,442
  7. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  8. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  9. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,845
  10. 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,546
  11. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,515
  12. 2017Deeper, Broader and Artier Domain Generalizationcited by 986
25 more
  1. 2017Deep Bayesian Active Learning with Image Datacited by 819
  2. 2017Zero-Shot Learning -- A Comprehensive Evaluation of the Good, the Bad and the Uglycited by 538
  3. 2019BatchBALD: Efficient and Diverse Batch Acquisition for Deep Bayesian Active Learningcited by 432
  4. 2017State-of-the-art Speech Recognition With Sequence-to-Sequence Modelscited by 196
  5. 2018Understanding Measures of Uncertainty for Adversarial Example Detectioncited by 177
  6. 2020A Review of Generalized Zero-Shot Learning Methodscited by 159
  7. 2022A Comparative Survey of Deep Active Learningcited by 89
  8. 2021Active Testing: Sample-Efficient Model Evaluationcited by 82
  9. 2014Minimax Analysis of Active Learningcited by 61
  10. 2023Learning to Maximize Mutual Information for Dynamic Feature Selectioncited by 56
  11. 2023Optimistic Active Exploration of Dynamical Systemscited by 32
  12. 2024Which LLM to Play? Convergence-Aware Online Model Selection with Time-Increasing Banditscited by 27
  13. 2020Online Active Model Selection for Pre-trained Classifierscited by 20
  14. 2020An Information-Theoretic Framework for Unifying Active Learning Problemscited by 14
  15. 2023Data pruning and neural scaling laws: fundamental limitations of score-based algorithmscited by 11
  16. 2013Approximation Algorithms for Stochastic Boolean Function Evaluation and Stochastic Submodular Set Covercited by 10
  17. 2023Likelihood Ratio Confidence Sets for Sequential Decision Makingcited by 10
  18. 2024Online Foundation Model Selection in Roboticscited by 8
  19. 2016Classifier Risk Estimation under Limited Labeling Resourcescited by 7
  20. 2024Advancing Deep Active Learning & Data Subset Selection: Unifying Principles with Information-Theory Intuitionscited by 7
  21. 2023Anytime Model Selection in Linear Banditscited by 6
  22. 2024On the Necessity of Collaboration for Online Model Selection with Decentralized Datacited by 6
  23. 2021Active Assessment of Prediction Services as Accuracy Surface Over Attribute Combinationscited by 5
  24. 2023Learning to Rank for Active Learning via Multi-Task Bilevel Optimizationcited by 3
  25. 2024Submodular Information Selection for Hypothesis Testing with Misclassification Penaltiescited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 5.

8 papers.

  1. 2025Consensus-Driven Active Model Selectioncited by 8
  2. 2025Estimating Time Series Foundation Model Transferability via In-Context Learningcited by 2
  3. 2026Large Language Model Selection with Limited Annotationscited by 1
  4. 2025Large Language Model Selection with Limited Annotationscited by 0
  5. 2026Candidate Comparability Before Promotion: Conditional Validation in Adaptive Network Intrusion Detectioncited by 0
  6. 2026How Many Labels Does Model Choice Need? Certificates and Budgets for Selective Predictioncited by 0
  7. 2026Teacher-Anchored Selection of Post-Training Quantized Models under Domain Shiftcited by 0
  8. 2026Which LLM to pick? Online Active Model Selection for Large Language Modelscited by 0