mlsanityData to 14 Oct 2026

LabelBench: A Comprehensive Framework for Benchmarking Adaptive Label-Efficient Learning

arXiv 2306.09910 ↗First posted 16 Jun 2023cs.LG cs.AI cs.CVcited by 20

Not on the map. Why?

Abstract (arXiv)

Labeled data are critical to modern machine learning applications, but obtaining labels can be expensive. To mitigate this cost, machine learning methods, such as transfer learning, semi-supervised learning and active learning, aim to be label-efficient: achieving high predictive performance from relatively few labeled examples. While obtaining the best label-efficiency in practice often requires combinations of these techniques, existing benchmark and evaluation frameworks do not capture a concerted combination of all such techniques. This paper addresses this deficiency by introducing LabelBench, a new computationally-efficient framework for joint evaluation of multiple label-efficient learning techniques. As an application of LabelBench, we introduce a novel benchmark of state-of-the-art active learning methods in combination with semi-supervised learning for fine-tuning pretrained vision transformers. Our benchmark demonstrates better label-efficiencies than previously reported in active learning. LabelBench's modular codebase is open-sourced for the broader community to contribute label-efficient learning methods and benchmarks. The repository can be found at: https://github.com/EfficientTraining/LabelBench.

Key facts. Cited by 20 papers in the map (data through 14 Oct 2026). Builds on 52 of its 76 references that are in the map. First posted on arXiv 16 Jun 2023.

Builds on

52 of 76 references.

References · 52

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  4. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  5. 2023Segment Anythingcited by 8,234
  6. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  7. 2020FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidencecited by 2,085
  8. 2020WILDS: A Benchmark of in-the-Wild Distribution Shiftscited by 1,375
  9. 2016Understanding intermediate layers using linear classifier probescited by 1,353
  10. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,274
  11. 2019Unsupervised Data Augmentation for Consistency Trainingcited by 1,060
  12. 2021Scaling Vision Transformerscited by 999
38 more
  1. 2021Florence: A New Foundation Model for Computer Visioncited by 804
  2. 2021Robust fine-tuning of zero-shot modelscited by 797
  3. 2019Big Transfer (BiT): General Visual Representation Learningcited by 769
  4. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 649
  5. 2021FILIP: Fine-grained Interactive Language-Image Pre-Trainingcited by 627
  6. 2019Deep Batch Active Learning by Diverse, Uncertain Gradient Lower Boundscited by 610
  7. 2020A Survey of Deep Active Learningcited by 554
  8. 2021FlexMatch: Boosting Semi-Supervised Learning with Curriculum Pseudo Labelingcited by 540
  9. 2022Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Taskscited by 511
  10. 2019Selection via Proxy: Efficient Data Selection for Deep Learningcited by 339
  11. 2017Functional Map of the Worldcited by 338
  12. 2022FreeMatch: Self-adaptive Thresholding for Semi-supervised Learningcited by 232
  13. 2022Active Learning on a Budget: Opposite Strategies Suit High and Low Budgetscited by 133
  14. 2023SoftMatch: Addressing the Quantity-Quality Trade-off in Semi-supervised Learningcited by 133
  15. 2021Batch Active Learning at Scalecited by 127
  16. 2020An Overview of Deep Semi-Supervised Learningcited by 121
  17. 2019Consistency-based Semi-supervised Active Learning: Towards Minimizing Labeling Costcited by 114
  18. 2021WRENCH: A Comprehensive Benchmark for Weak Supervisioncited by 96
  19. 2022USB: A Unified Semi-supervised Learning Benchmark for Classificationcited by 95
  20. 2022A Comparative Survey of Deep Active Learningcited by 89
  21. 2021Gone Fishing: Neural Active Learning with Fisher Embeddingscited by 88
  22. 2021The iWildCam 2021 Competition Datasetcited by 77
  23. 2022SVFormer: Semi-supervised Video Transformer for Action Recognitioncited by 68
  24. 2022Active Learning Through a Covering Lenscited by 66
  25. 2021Semi-Supervised Active Learning with Temporal Output Discrepancycited by 50
  26. 2022Semi-supervised Vision Transformers at Scalecited by 49
  27. 2019Rethinking deep active learning: Using unlabeled data at model trainingcited by 39
  28. 2019Parting with Illusions about Deep Active Learningcited by 38
  29. 2021Effective Evaluation of Deep Active Learning on Image Classification Taskscited by 37
  30. 2022Active Learning Helps Pretrained Models Learn the Intended Taskcited by 33
  31. 2022GALAXY: Graph-based Active Learning at the Extremecited by 29
  32. 2021Active Learning at the ImageNet Scalecited by 28
  33. 2020Similarity Search for Efficient Active Learning and Search of Rare Conceptscited by 24
  34. 2020On the Marginal Benefit of Active Learning: Does Self-Supervision Eat Its Cake?cited by 18
  35. 2020Semi-supervised Batch Active Learning via Bilevel Optimizationcited by 16
  36. 2022Active Learning with Expected Error Reductioncited by 9
  37. 2023Accelerating Batch Active Learning Using Continual Learning Techniquescited by 8
  38. 2023Transfer Learning for Fine-grained Classification Using Semi-supervised Learning and Visual Transformerscited by 5

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 7; 2025: 9; 2026: 3.

20 papers.

Sort:

By area:

  1. 2024An Experimental Design Framework for Label-Efficient Supervised Finetuning of Large Language Modelscited by 23
  2. 2024A Cross-Domain Benchmark for Active Learningcited by 12
  3. 2024Making Better Use of Unlabelled Data in Bayesian Active Learningcited by 10
  4. 2023Improved Algorithm for Deep Active Learning under Imbalance via Optimal Separationcited by 8
  5. 2024AHA: Human-Assisted Out-of-Distribution Generalization and Detectioncited by 7
  6. 2025Test-Time Matching: Unlocking Compositional Reasoning in Multimodal Modelscited by 6
  7. 2024Deep Active Learning in the Open Worldcited by 5
  8. 2026Active Testing of Large Language Models via Approximate Neyman Allocationcited by 5
  9. 2024Feature Alignment: Rethinking Efficient Active Learning via Proxy in the Context of Pre-trained Modelscited by 4
  10. 2025Enhancing Cost Efficiency in Active Learning with Candidate Set Querycited by 3
  11. 2025Active Learning with a Noisy Annotatorcited by 3
  12. 2025Towards Multimodal Active Learning: Efficient Learning with Limited Paired Datacited by 3
8 more
  1. 2024COBRA: COmBinatorial Retrieval Augmentation for Few-Shot Adaptationcited by 2
  2. 2025Mixtraining: A Better Trade-Off Between Compute and Performancecited by 2
  3. 2025To Label or Not to Label: PALM -- A Predictive Model for Evaluating Sample Efficiency in Active Learning Modelscited by 2
  4. 2025No Free Lunch in Active Learning: LLM Embedding Quality Dictates Query Strategy Successcited by 1
  5. 2025Improving Task Diversity in Label Efficient Supervised Finetuning of LLMscited by 1
  6. 2026Online Finetuning Decision Transformers with Pure RL Gradientscited by 1
  7. 2025Batch Bayesian Active Learning with Partial Batch Label Samplingcited by 0
  8. 2026Active Learning with Foundation Model Priors: Efficient Learning under Class Imbalancecited by 0