mlsanityData to 14 Oct 2026

How Many Validation Labels Do You Need? Exploring the Design Space of Label-Efficient Model Ranking

arXiv 2312.01619 ↗First posted 4 Dec 2023cs.LGcited by 8

Not on the map. Why?

Abstract (arXiv)

This paper presents LEMR (Label-Efficient Model Ranking) and introduces the MoraBench Benchmark. LEMR is a novel framework that minimizes the need for costly annotations in model selection by strategically annotating instances from an unlabeled validation set. To evaluate LEMR, we leverage the MoraBench Benchmark, a comprehensive collection of model outputs across diverse scenarios. Our extensive evaluation across 23 different NLP tasks in semi-supervised learning, weak supervision, and prompt selection tasks demonstrates LEMR's effectiveness in significantly reducing labeling costs. Key findings highlight the impact of suitable ensemble methods, uncertainty sampling strategies, and model committee selection in enhancing model ranking accuracy. LEMR, supported by the insights from MoraBench, provides a cost-effective and accurate solution for model selection, especially valuable in resource-constrained environments.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Builds on 43 of its 79 references that are in the map. First posted on arXiv 4 Dec 2023.

Builds on

43 of 79 references.

References · 43

  1. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  2. 2015Character-level Convolutional Networks for Text Classificationcited by 2,330
  3. 2020FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidencecited by 2,085
  4. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,470
  5. 2019MixMatch: A Holistic Approach to Semi-Supervised Learningcited by 1,391
  6. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,135
  7. 2019Unsupervised Data Augmentation for Consistency Trainingcited by 1,060
  8. 2017Virtual Adversarial Training: A Regularization Method for Supervised and Semi-Supervised Learningcited by 920
  9. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 896
  10. 2019Adversarial NLI: A New Benchmark for Natural Language Understandingcited by 807
  11. 2021FlexMatch: Boosting Semi-Supervised Learning with Curriculum Pseudo Labelingcited by 540
  12. 2018WiC: the Word-in-Context Dataset for Evaluating Context-Sensitive Meaning Representationscited by 372
31 more
  1. 2019ReMixMatch: Semi-Supervised Learning with Distribution Alignment and Augmentation Anchoringcited by 349
  2. 2021True Few-Shot Learning with Language Modelscited by 344
  3. 2015Semi-Supervised Learning with Ladder Networkscited by 247
  4. 2014Learning with Pseudo-Ensemblescited by 241
  5. 2023Large Language Model as Attributed Training Data Generator: A Tale of Diversity and Biascited by 232
  6. 2020CoMatch: Semi-supervised Learning with Contrastive Graph Regularizationcited by 167
  7. 2021Dash: Semi-Supervised Learning with Dynamic Thresholdingcited by 148
  8. 2022SimMatch: Semi-supervised Learning with Similarity Matchingcited by 118
  9. 2021AdaMatch: A Unified Approach to Semi-Supervised Learning and Domain Adaptationcited by 104
  10. 2021WRENCH: A Comprehensive Benchmark for Weak Supervisioncited by 96
  11. 2018The Description Length of Deep Learning Modelscited by 72
  12. 2021FLEX: Unifying Evaluation for Few-Shot NLPcited by 67
  13. 2022Prompt Consistency for Zero-Shot Task Generalizationcited by 67
  14. 2021Revisiting Uncertainty-based Query Strategies for Active Learning with Transformerscited by 60
  15. 2020Learning from Rules Generalizing Labeled Exemplarscited by 59
  16. 2017Learning the Structure of Generative Models without Labeled Datacited by 50
  17. 2020Denoising Multi-Source Weak Supervision for Neural Text Classificationcited by 42
  18. 2023Cost-Effective Hyperparameter Optimization for Large Language Model Generation Inferencecited by 40
  19. 2017Early Stopping without a Validation Setcited by 39
  20. 2021Revisiting Self-Training for Few-Shot Learning of Language Modelcited by 23
  21. 2022Learning Hyper Label Model for Programmatic Weak Supervisioncited by 16
  22. 2022Cold-Start Data Selection for Few-shot Language Model Fine-tuning: A Prompt-Based Uncertainty Propagation Approachcited by 16
  23. 2021Creating Training Sets via Weak Indirect Supervisioncited by 14
  24. 2023Neighborhood-Regularized Self-Training for Learning with Few Labelscited by 14
  25. 2019DDPNAS: Efficient Neural Architecture Search via Dynamic Distribution Pruningcited by 13
  26. 2021An Empirical Study on Hyperparameter Optimization for Fine-Tuning Pre-trained Language Modelscited by 12
  27. 2023Fair Enough: Standardizing Evaluation and Model Selection for Fairness Research in NLPcited by 11
  28. 2021Disentangling Sampling and Labeling Bias for Learning in Large-Output Spacescited by 9
  29. 2023HyperTime: Hyperparameter Optimization for Combating Temporal Distribution Shiftscited by 8
  30. 2022Mining Robust Default Configurations for Resource-constrained AutoMLcited by 2
  31. 2022ACE: Adaptive Constraint-aware Early Stopping in Hyperparameter Optimizationcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 5; 2025: 2; 2026: 1.

8 papers.

  1. 2024FlipAttack: Jailbreak LLMs via Flippingcited by 80
  2. 2025The Hallucination Tax of Reinforcement Finetuningcited by 40
  3. 2024Residual-based Language Models are Free Boosters for Biomedical Imagingcited by 24
  4. 2024An Experimental Design Framework for Label-Efficient Supervised Finetuning of Large Language Modelscited by 23
  5. 2024Adaptive Ensembles of Fine-Tuned Transformers for LLM-Generated Text Detectioncited by 22
  6. 2024The New Agronomists: Language Models are Experts in Crop Managementcited by 13
  7. 2025DISCO: Diversifying Sample Condensation for Efficient Model Evaluationcited by 5
  8. 2026When the Strongest Teacher Is Not the Best Teacher: Student-Centric Answer Selectioncited by 1