mlsanityData to 14 Oct 2026

LaFTer: Label-Free Tuning of Zero-shot Classifier using Language and Unlabeled Image Collections

arXiv 2305.18287 ↗First posted 29 May 2023cs.CV cs.CLcited by 43

Not on the map. Why?

Cited by 22 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Recently, large-scale pre-trained Vision and Language (VL) models have set a new state-of-the-art (SOTA) in zero-shot visual classification enabling open-vocabulary recognition of potentially unlimited set of categories defined as simple language prompts. However, despite these great advances, the performance of these zeroshot classifiers still falls short of the results of dedicated (closed category set) classifiers trained with supervised fine tuning. In this paper we show, for the first time, how to reduce this gap without any labels and without any paired VL data, using an unlabeled image collection and a set of texts auto-generated using a Large Language Model (LLM) describing the categories of interest and effectively substituting labeled visual instances of those categories. Using our label-free approach, we are able to attain significant performance improvements over the zero-shot performance of the base VL model and other contemporary methods and baselines on a wide variety of datasets, demonstrating absolute improvement of up to 11.7% (3.8% on average) in the label-free setting. Moreover, despite our approach being label-free, we observe 1.3% average gains over leading few-shot prompting baselines that do use 5-shot supervision.

Key facts. Cited by 43 papers in the map (data through 14 Oct 2026). Builds on 42 of its 60 references that are in the map. First posted on arXiv 29 May 2023.

Builds on

42 of 60 references.

References · 42

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,475
  4. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  5. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  6. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  7. 2020Exploring Simple Siamese Representation Learningcited by 2,866
  8. 2021Learning to Prompt for Vision-Language Modelscited by 2,281
  9. 2020FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidencecited by 2,085
  10. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,850
  11. 2022Visual Prompt Tuningcited by 1,681
  12. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,619
30 more
  1. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,553
  2. 2022Conditional Prompt Learning for Vision-Language Modelscited by 1,433
  3. 2019Natural Adversarial Examplescited by 1,408
  4. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,274
  5. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,230
  6. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,214
  7. 2019Unsupervised Data Augmentation for Consistency Trainingcited by 1,060
  8. 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapterscited by 1,026
  9. 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,013
  10. 2022A Generalist Agentcited by 808
  11. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 649
  12. 2021FILIP: Fine-grained Interactive Language-Image Pre-Trainingcited by 627
  13. 2022Efficient Test-Time Model Adaptation without Forgettingcited by 469
  14. 2021SLIP: Self-supervision meets Language-Image Pre-trainingcited by 436
  15. 2021Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modelingcited by 431
  16. 2021Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigmcited by 405
  17. 2022Test-Time Prompt Tuning for Zero-Shot Generalization in Vision-Language Modelscited by 382
  18. 2021Factual Probing Is [MASK]: Learning vs. Learning to Recallcited by 291
  19. 2022What does a platypus look like? Generating customized prompts for zero-shot image classificationcited by 267
  20. 2015Semi-Supervised Learning with Ladder Networkscited by 247
  21. 2022Prompt Distribution Learningcited by 235
  22. 2022Vision-Language Pre-Training with Triple Contrastive Learningcited by 212
  23. 2022Unsupervised Prompt Learning for Vision-Language Modelscited by 127
  24. 2022CyCLIP: Cyclic Contrastive Language-Image Pretrainingcited by 127
  25. 2021Semi-Supervised Learning of Visual Features by Non-Parametrically Predicting View Assignments with Support Samplescited by 114
  26. 2022PyramidCLIP: Hierarchical Feature Alignment for Vision-language Model Pretrainingcited by 104
  27. 2021CLOOB: Modern Hopfield Networks with InfoLOOB Outperform CLIPcited by 89
  28. 2022ActMAD: Activation Matching to Align Distributions for Test-Time-Trainingcited by 41
  29. 2021Data Efficient Language-supervised Zero-shot Recognition with Optimal Transport Distillationcited by 36
  30. 2022Improving Zero-Shot Models with Label Distribution Priorscited by 14

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 3; 2024: 20; 2025: 14; 2026: 6.

43 papers.

Sort:

By area:

  1. 2024PromptKD: Unsupervised Prompt Distillation for Vision-Language Modelscited by 72
  2. 2024Improved Zero-Shot Classification by Adapting VLMs with Text Descriptionscited by 54
  3. 2024Meta-Prompting for Automating Zero-shot Visual Recognition with LLMscited by 29
  4. 2024Towards Multimodal In-Context Learning for Vision & Language Modelscited by 25
  5. 2024Candidate Pseudolabel Learning: Enhancing Vision-Language Models by Prompt Tuning with Unlabeled Datacited by 20
  6. 2024Cascade Prompt Learning for Vision-Language Model Adaptationcited by 19
  7. 2023PerceptionCLIP: Visual Classification by Inferring and Conditioning on Contextscited by 17
  8. 2025CLIP-Powered Domain Generalization and Domain Adaptation: A Comprehensive Surveycited by 15
  9. 2023Follow-Up Differential Descriptions: Language Models Resolve Ambiguities for Image Classificationcited by 13
  10. 2025TTRV: Test-Time Reinforcement Learning for Vision Language Modelscited by 10
  11. 2023TAP: Targeted Prompting for Task Adaptive Generation of Textual Training Instances for Visual Classificationcited by 8
  12. 2024Training-Free Semantic Segmentation via LLM-Supervisioncited by 8
31 more
  1. 2024SenCLIP: Enhancing zero-shot land-use mapping for Sentinel-2 with ground-level promptingcited by 7
  2. 2024GLOV: Guided Large Language Models as Implicit Optimizers for Vision Language Modelscited by 6
  3. 2024Teaching VLMs to Localize Specific Objects from In-context Examplescited by 6
  4. 2024On the Element-Wise Representation and Reasoning in Zero-Shot Image Recognition: A Systematic Surveycited by 5
  5. 2024DPA: Dual Prototypes Alignment for Unsupervised Adaptation of Vision-Language Modelscited by 5
  6. 2024CLIP meets DINO for Tuning Zero-Shot Classifier using Unlabeled Image Collectionscited by 5
  7. 2024I can listen but cannot read: An evaluation of two-tower multimodal systems for instrument recognitioncited by 4
  8. 2025VisualOverload: Probing Visual Understanding of VLMs in Really Dense Scenescited by 4
  9. 2024A sound description: Exploring prompt templates and class descriptions to enhance zero-shot audio classificationcited by 3
  10. 2024Embracing Diversity: Interpretable Zero-shot classification beyond one vector per classcited by 2
  11. 2024Can language-guided unsupervised adaptation improve medical image classification using unpaired images and texts?cited by 2
  12. 2024Cross-Modal Few-Shot Learning: a Generative Transfer Learning Frameworkcited by 2
  13. 2025Generate, Transduct, Adapt: Iterative Transduction with VLMscited by 2
  14. 2025DesCLIP: Robust Continual Learning via General Attribute Descriptions for VLM-Based Visual Recognitioncited by 2
  15. 2024XDT-CXR: Investigating Cross-Disease Transferability in Zero-Shot Binary Classification of Chest X-Rayscited by 1
  16. 2025Towards Fine-Grained Adaptation of CLIP via a Self-Trained Alignment Scorecited by 1
  17. 2025Slot Attention-based Feature Filtering for Few-Shot Learningcited by 1
  18. 2026Multiview Self-Representation Learning across Heterogeneous Viewscited by 1
  19. 2024Textual Training for the Hassle-Free Removal of Unwanted Visual Data: Case Studies on OOD and Hateful Image Detectioncited by 0
  20. 2025Reducing Annotation Burden in Physical Activity Research Using Vision-Language Modelscited by 0
  21. 2025GeoVision Labeler: Zero-Shot Geospatial Classification with Vision and Language Modelscited by 0
  22. 2025Prototype-Guided Pseudo-Labeling with Neighborhood-Aware Consistency for Unsupervised Adaptationcited by 0
  23. 2025Stochastic-based Patch Filtering for Few-Shot Learningcited by 0
  24. 2025No Labels Needed: Zero-Shot Image Classification with Collaborative Self-Learningcited by 0
  25. 2025microCLIP: Unsupervised CLIP Adaptation via Coarse-Fine Token Fusion for Fine-Grained Image Classificationcited by 0
  26. 2025Annotation-Free Class-Incremental Learningcited by 0
  27. 2026AdaDINO: Context-Adaptive DINO-Distilled Vision Foundation Models for Efficient Open-Vocabulary Edge Inferencecited by 0
  28. 2026Learning Subspace-Preserving Sparse Attention Graphs from Heterogeneous Multiview Datacited by 0
  29. 2026One Stone, Three Birds: Self-adaptive Optimal Transport for Multi-VLM Selection, Adaptation, and Ensemblingcited by 0
  30. 2026The Geometry of Saturation: Effective Rank Predicts When Labels Stop Helping in Few-Shot Classificationcited by 0
  31. 2026Dual-Stream Semantic Guidance with Prototype Anchor Calibration for Source-Fully-Free Adaptation of Vision-Language Modelscited by 0