Self-Supervised Open-Ended Classification with Small Visual Language Models
On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP
Abstract (arXiv)
We present Self-Context Adaptation (SeCAt), a self-supervised approach that unlocks few-shot abilities for open-ended classification with small visual language models. Our approach imitates image captions in a self-supervised way based on clustering a large pool of images followed by assigning semantically-unrelated names to clusters. By doing so, we construct a training signal consisting of interleaved sequences of image and pseudocaption pairs and a query image, which we denote as the 'self-context' sequence. Based on this signal the model is trained to produce the right pseudo-caption. We demonstrate the performance and flexibility of SeCAt on several multimodal few-shot datasets, spanning various granularities. By using models with approximately 1B parameters we outperform the few-shot abilities of much larger models, such as Frozen and FROMAGe. SeCAt opens new possibilities for research and applications in open-ended few-shot learning that otherwise requires access to large or proprietary models.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 43 of its 60 references that are in the map. First posted on arXiv 30 Sep 2023.
Builds on
43 of 60 references.
References · 43
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
- 2020Language Models are Few-Shot Learnerscited by 32,419
- 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
- 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
- 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
- 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
- 2021Zero-Shot Text-to-Image Generationcited by 4,288
- 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
- 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
- 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
- 2022Training Compute-Optimal Large Language Modelscited by 3,019
- 2022Emergent Abilities of Large Language Modelscited by 2,512
31 more
- 2017Billion-scale similarity search with GPUscited by 2,497
- 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
- 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,850
- 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,202
- 2021What Makes Good In-Context Examples for GPT-3?cited by 1,195
- 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,135
- 2018Deep Clustering for Unsupervised Learning of Visual Featurescited by 1,028
- 2015Unsupervised Deep Embedding for Clustering Analysiscited by 767
- 2016TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systemscited by 664
- 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 661
- 2021ClipCap: CLIP Prefix for Image Captioningcited by 559
- 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 559
- 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
- 2019Self-labelling via simultaneous clustering and representation learningcited by 528
- 2022Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Taskscited by 511
- 2018Invariant Information Clustering for Unsupervised Image Classification and Segmentationcited by 455
- 2021MetaICL: Learning to Learn In Contextcited by 454
- 2023Larger language models do in-context learning differentlycited by 394
- 2020SCAN: Learning to Classify Images without Labelscited by 352
- 2023Multimodal C4: An Open, Billion-scale Corpus of Images Interleaved with Textcited by 208
- 2016Joint Unsupervised Learning of Deep Representations and Image Clusterscited by 205
- 2022Re3: Generating Longer Stories With Recursive Reprompting and Revisioncited by 193
- 2022Deep Spectral Methods: A Surprisingly Strong Baseline for Unsupervised Semantic Segmentation and Localizationcited by 150
- 2020Labelling unlabelled videos from scratch with multi-modal self-supervisioncited by 100
- 2017Unsupervised Learning by Predicting Noisecited by 98
- 2022Language Models are General-Purpose Interfacescited by 97
- 2018Boosting Self-Supervised Learning via Knowledge Transfercited by 92
- 2023Symbol tuning improves in-context learning in language modelscited by 75
- 2022Transcending Scaling Laws with 0.1% Extra Computecited by 57
- 2023Meta Learning to Bridge Vision and Language Models for Multimodal Few-Shot Learningcited by 20
- 2021Motion-Augmented Self-Training for Video Recognition at Smaller Scalecited by 17
Built on by
Citing papers per year: 2025: 1; 2026: 1.
2 papers.
Similar, not cited
Close in wording, no citation either way
- 2024Efficient Test-Time Prompt Tuning for Vision-Language Models
- 2023SILC: Improving Vision Language Pretraining with Self-Distillation
- 2024Image-Caption Encoding for Improving Zero-Shot Generalization
- 2025Enhancing Multimodal In-Context Learning for Image Classification through Coreset Optimization
- 2023Towards Realistic Zero-Shot Classification via Self Structural Semantic Alignment
- 2025Solving Semi-Supervised Few-Shot Learning from an Auto-Annotation Perspective
- 2023ECO: Ensembling Context Optimization for Vision-Language Models
- 2024Few-Shot Recognition via Stage-Wise Retrieval-Augmented Finetuning