mlsanityData to 7 Oct 2026

Selective Vision-Language Subspace Projection for Few-shot CLIP

arXiv 2407.16977 ↗First posted 24 Jul 2024cs.CV cs.MMcited by 18

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

Vision-language models such as CLIP are capable of mapping the different modality data into a unified feature space, enabling zero/few-shot inference by measuring the similarity of given images and texts. However, most existing methods overlook modality gaps in CLIP's encoded features, which is shown as the text and image features lie far apart from each other, resulting in limited classification performance. To tackle this issue, we introduce a method called Selective Vision-Language Subspace Projection (SSP), which incorporates local image features and utilizes them as a bridge to enhance the alignment between image-text pairs. Specifically, our SSP framework comprises two parallel modules: a vision projector and a language projector. Both projectors utilize local image features to span the respective subspaces for image and texts, thereby projecting the image and text features into their respective subspaces to achieve alignment. Moreover, our approach entails only training-free matrix calculations and can be seamlessly integrated into advanced CLIP-based few-shot learning frameworks. Extensive experiments on 11 datasets have demonstrated SSP's superior text-image alignment capabilities, outperforming the state-of-the-art alignment methods. The code is available at https://github.com/zhuhsingyuu/SSP

Key facts. Cited by 18 papers in the map (data through 7 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 28 of its 65 references that are in the map. First posted on arXiv 24 Jul 2024.

Builds on

28 of 65 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023Debiased Fine-Tuning for Vision-language Models by Prompt Regularization29 Jan
  2. 2022Tip-Adapter: Training-free Adaption of CLIP for Few-shot Classification19 Jul
  3. 2022Prompt-aligned Gradient for Prompt Tuning30 May
  4. 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapters9 Oct

Other references · 24

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  5. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  6. 2013Describing Textures in the Wildcited by 2,110
  7. 2013Fine-Grained Visual Classification of Aircraftcited by 1,701
  8. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  9. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,507
  10. 2019Natural Adversarial Examplescited by 1,401
  11. 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,010
  12. 2021Open-vocabulary Object Detection via Vision and Language Knowledge Distillationcited by 903
12 more
  1. 2023Vision-Language Models for Vision Tasks: A Surveycited by 834
  2. 2021ClipCap: CLIP Prefix for Image Captioningcited by 557
  3. 2021PointCLIP: Point Cloud Understanding by CLIPcited by 461
  4. 2021AudioCLIP: Extending CLIP to Image, Text and Audiocited by 335
  5. 2023Not All Features Matter: Enhancing Few-shot CLIP with Adaptive Prior Refinementcited by 84
  6. 2023GraphAdapter: Tuning Vision-Language Models With Dual Knowledge Graphcited by 69
  7. 2024Probabilistic Contrastive Learning for Long-Tailed Visual Recognitioncited by 42
  8. 2022Cross-Modal Adapter for Vision-Language Retrievalcited by 35
  9. 2023Generalized Logit Adjustment: Calibrating Fine-tuned Models by Removing Label Bias in Foundation Modelscited by 32
  10. 2024Few-shot Adaptation of Multi-modal Foundation Models: A Surveycited by 28
  11. 2023CLAPSpeech: Learning Prosody from Text Context with Contrastive Language-Audio Pre-trainingcited by 17
  12. 2024Mind the Modality Gap: Towards a Remote Sensing Vision-Language Model via Cross-modal Alignmentcited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 13; 2026: 3.

18 papers.

Sort:

By area:

  1. 2025On the Generalization of SFT: A Reinforcement Learning Perspective with Reward Rectificationcited by 121
  2. 2024LLaVA Steering: Visual Instruction Tuning with 500x Fewer Parameters through Modality Linear Representation-Steeringcited by 50
  3. 2024Enhancing Zero-Shot Vision Models by Label-Free Prompt Distribution Learning and Bias Correctingcited by 26
  4. 2025Accelerating Diffusion Transformer via Error-Optimized Cachecited by 14
  5. 2025Enhancing CLIP Robustness via Cross-Modality Alignmentcited by 14
  6. 2025Dynamic Multimodal Prototype Learning in Vision-Language Modelscited by 13
  7. 2025Advancing Reliable Test-Time Adaptation of Vision-Language Models under Visual Variationscited by 9
  8. 2025Real-Time Motion-Controllable Autoregressive Video Diffusioncited by 7
  9. 2026Look Carefully: Adaptive Visual Reinforcements in Multimodal Large Language Models for Hallucination Mitigationcited by 7
  10. 2025ACE: Concept Editing in Diffusion Models without Performance Degradationcited by 6
  11. 2025Distill CLIP (DCLIP): Enhancing Image-Text Retrieval via Cross-Modal Transformer Distillationcited by 6
  12. 2025Streaming Drag-Oriented Interactive Video Manipulation: Drag Anything, Anytime!cited by 5
6 more
  1. 2025Hierarchical Semantic Alignment for Image Clusteringcited by 5
  2. 2025vMFCoOp: Towards Equilibrium on a Unified Hyperspherical Manifold for Prompting Biomedical VLMscited by 3
  3. 2026Revisiting Robustness for LLM Safety Alignment via Selective Geometry Controlcited by 3
  4. 2025Does CLIP perceive art the same way we do?cited by 2
  5. 2025Prompt Optimization Meets Subspace Representation Learning for Few-shot Out-of-Distribution Detectioncited by 0
  6. 2026Drift-Augmented Scoring: Text-Derived Noise Robustness for Zero-Shot Audio-Language Classificationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Post-pre-training for Modality Alignment in Vision-Language Foundation Models
  2. 2024Visual-Text Cross Alignment: Refining the Similarity Score in Vision-Language Models
  3. 2025Kernel-based Unsupervised Embedding Alignment for Enhanced Visual Representation in Vision-language Models
  4. 2025PowerCLIP: Powerset Alignment for Contrastive Pre-Training
  5. 2024S3: Synonymous Semantic Space for Improving Zero-Shot Generalization of Vision-Language Models
  6. 2024Finetuning CLIP to Reason about Pairwise Differences
  7. 2024Learning to Rank Pre-trained Vision-Language Models for Downstream Tasks
  8. 2026Cross-Modal Prototype Alignment and Mixing for Training-Free Few-Shot Classification