mlsanityData to 14 Oct 2026

Supervised Classification Heads as Semantic Prototypes: Unlocking Vision-Language Alignment via Weight Recycling

arXiv 2605.22484 ↗First posted 21 May 2026cs.CVcited by 0

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

Vision-Language Models (VLMs) excel at tasks like zero-shot classification and cross-modal retrieval by mapping images and text to a shared space, but this requires expensive end-to-end training with massive paired datasets. Current post-hoc alignment methods reduce computational costs by connecting pretrained encoders through lightweight mappings, yet still demand substantial paired data. In this work, we investigate the potential of repurposing the classification heads of pretrained vision models as semantic prototypes. The recycling of these weights, typically discarded after pretraining, unlocks two distinct capabilities: it enables zero-shot alignment by using weights as semantic anchors, and serves as a robust data augmentation strategy by mixing these prototypes with real image-text pairs. We demonstrate that integrating our approach with several state-of-the-art post-hoc alignment techniques consistently boosts accuracy in cross-modal retrieval, zero- and few-shot classification tasks.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 31 of its 47 references that are in the map. First posted on arXiv 21 May 2026.

Builds on

31 of 47 references.

Closest ancestors · 2

Same direction, newest first

  1. 2022Conditional Prompt Learning for Vision-Language Models10 Mar
  2. 2021Learning to Prompt for Vision-Language Models2 Sep

Other references · 29

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  4. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,045
  5. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  6. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,375
  7. 2013Describing Textures in the Wildcited by 2,125
  8. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,362
  9. 2020MPNet: Masked and Permuted Pre-training for Language Understandingcited by 981
  10. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 968
  11. 2017Remote Sensing Image Scene Classification: Benchmark and State of the Artcited by 797
  12. 2020Prevalence of Neural Collapse during the terminal phase of deep learning trainingcited by 719
17 more
  1. 2021ImageNet-21K Pretraining for the Massescited by 613
  2. 2021LiT: Zero-Shot Transfer with Locked-image text Tuningcited by 577
  3. 2022Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learningcited by 561
  4. 2023Demystifying CLIP Datacited by 270
  5. 2022CLIP Models are Few-shot Learners: Empirical Studies on VQA and Visual Entailmentcited by 113
  6. 2022Inducing Neural Collapse in Imbalanced Learning: Do We Really Need a Learnable Classifier at the End of Deep Neural Network?cited by 100
  7. 2023DeCap: Decoding CLIP Latents for Zero-Shot Captioning via Text-Only Trainingcited by 84
  8. 2023Text-To-Concept (and Back) via Cross-Model Alignmentcited by 64
  9. 2022ASIF: Coupled Data Turns Unimodal Models to Multimodal Without Trainingcited by 47
  10. 2023A Closer Look at Few-shot Classification Againcited by 46
  11. 2024Do Vision and Language Encoders Represent the World Similarly?cited by 44
  12. 2023Generalized Neural Collapse for a Large Number of Classescited by 39
  13. 2024Assessing and Learning Alignment of Unimodal Vision and Language Modelscited by 32
  14. 2023Image-free Classifier Injection for Zero-Shot Classificationcited by 15
  15. 2022Deep transfer learning for image classification: a surveycited by 13
  16. 2024CSA: Data-efficient Mapping of Unimodal Features to Multimodal Featurescited by 12
  17. 2024FLAME: Frozen Large Language Models Enable Data-Efficient Language-Image Pre-trainingcited by 12

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025TinyAlign: Boosting Lightweight Vision-Language Models by Mitigating Modal Alignment Bottlenecks
  2. 2026Explaining, Verifying, and Aligning Semantic Hierarchies in Vision-Language Model Embeddings
  3. 2026Cross-Modal Prototype Alignment and Mixing for Training-Free Few-Shot Classification
  4. 2022Single-Stream Multi-Level Alignment for Vision-Language Pretraining
  5. 2022PyramidCLIP: Hierarchical Feature Alignment for Vision-language Model Pretraining
  6. 2022MVPTR: Multi-Level Semantic Alignment for Vision-Language Pre-Training via Multi-Stage Learning
  7. 2023Towards Realistic Zero-Shot Classification via Self Structural Semantic Alignment
  8. 2025VL-SAE: Interpreting and Enhancing Vision-Language Alignment with a Unified Concept Set