# Selective Vision-Language Subspace Projection for Few-shot CLIP

arXiv: https://arxiv.org/abs/2407.16977 · HTML: https://mlsanity.com/paper/2407.16977/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 28 of its 65 references that are in the map.
- First posted on arXiv 24 Jul 2024.

## Placed in

- Multimodal LLMs / vision-language models / [Prompt learning and adaptation of CLIP](https://mlsanity.com/mllm/prompt-learning-and-adaptation-of-clip/)

## Builds on (most cited first, 28 of 28)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://mlsanity.com/paper/2201.12086/) (2022)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [Describing Textures in the Wild](https://mlsanity.com/paper/1311.3618/) (2013)
- [Fine-Grained Visual Classification of Aircraft](https://mlsanity.com/paper/1306.5151/) (2013)
- [Align before Fuse: Vision and Language Representation Learning with Momentum Distillation](https://mlsanity.com/paper/2107.07651/) (2021)
- [Do ImageNet Classifiers Generalize to ImageNet?](https://mlsanity.com/paper/1902.10811/) (2019)
- [Natural Adversarial Examples](https://mlsanity.com/paper/1907.07174/) (2019)
- [CLIP-Adapter: Better Vision-Language Models with Feature Adapters](https://mlsanity.com/paper/2110.04544/) (2021)
- [Learning Robust Global Representations by Penalizing Local Predictive Power](https://mlsanity.com/paper/1905.13549/) (2019)
- [Open-vocabulary Object Detection via Vision and Language Knowledge Distillation](https://mlsanity.com/paper/2104.13921/) (2021)
- [Vision-Language Models for Vision Tasks: A Survey](https://mlsanity.com/paper/2304.00685/) (2023)
- [ClipCap: CLIP Prefix for Image Captioning](https://mlsanity.com/paper/2111.09734/) (2021)
- [PointCLIP: Point Cloud Understanding by CLIP](https://mlsanity.com/paper/2112.02413/) (2021)
- [Tip-Adapter: Training-free Adaption of CLIP for Few-shot Classification](https://mlsanity.com/paper/2207.09519/) (2022)
- [Prompt-aligned Gradient for Prompt Tuning](https://mlsanity.com/paper/2205.14865/) (2022)
- [AudioCLIP: Extending CLIP to Image, Text and Audio](https://mlsanity.com/paper/2106.13043/) (2021)
- [Not All Features Matter: Enhancing Few-shot CLIP with Adaptive Prior Refinement](https://mlsanity.com/paper/2304.01195/) (2023)
- [GraphAdapter: Tuning Vision-Language Models With Dual Knowledge Graph](https://mlsanity.com/paper/2309.13625/) (2023)
- [Probabilistic Contrastive Learning for Long-Tailed Visual Recognition](https://mlsanity.com/paper/2403.06726/) (2024)
- [Cross-Modal Adapter for Vision-Language Retrieval](https://mlsanity.com/paper/2211.09623/) (2022)
- [Generalized Logit Adjustment: Calibrating Fine-tuned Models by Removing Label Bias in Foundation Models](https://mlsanity.com/paper/2310.08106/) (2023)
- [Few-shot Adaptation of Multi-modal Foundation Models: A Survey](https://mlsanity.com/paper/2401.01736/) (2024)
- [Debiased Fine-Tuning for Vision-language Models by Prompt Regularization](https://mlsanity.com/paper/2301.12429/) (2023)
- [CLAPSpeech: Learning Prosody from Text Context with Contrastive Language-Audio Pre-training](https://mlsanity.com/paper/2305.10763/) (2023)
- [Mind the Modality Gap: Towards a Remote Sensing Vision-Language Model via Cross-modal Alignment](https://mlsanity.com/paper/2402.09816/) (2024)

## Built on by (most cited first, 18 of 18)

- [On the Generalization of SFT: A Reinforcement Learning Perspective with Reward Rectification](https://mlsanity.com/paper/2508.05629/) (2025)
- [LLaVA Steering: Visual Instruction Tuning with 500x Fewer Parameters through Modality Linear Representation-Steering](https://mlsanity.com/paper/2412.12359/) (2024)
- [Enhancing Zero-Shot Vision Models by Label-Free Prompt Distribution Learning and Bias Correcting](https://mlsanity.com/paper/2410.19294/) (2024)
- [Enhancing CLIP Robustness via Cross-Modality Alignment](https://mlsanity.com/paper/2510.24038/) (2025)
- [Accelerating Diffusion Transformer via Error-Optimized Cache](https://mlsanity.com/paper/2501.19243/) (2025)
- [Dynamic Multimodal Prototype Learning in Vision-Language Models](https://mlsanity.com/paper/2507.03657/) (2025)
- [Advancing Reliable Test-Time Adaptation of Vision-Language Models under Visual Variations](https://mlsanity.com/paper/2507.09500/) (2025)
- [Real-Time Motion-Controllable Autoregressive Video Diffusion](https://mlsanity.com/paper/2510.08131/) (2025)
- [Look Carefully: Adaptive Visual Reinforcements in Multimodal Large Language Models for Hallucination Mitigation](https://mlsanity.com/paper/2602.24041/) (2026)
- [ACE: Concept Editing in Diffusion Models without Performance Degradation](https://mlsanity.com/paper/2503.08116/) (2025)
- [Distill CLIP (DCLIP): Enhancing Image-Text Retrieval via Cross-Modal Transformer Distillation](https://mlsanity.com/paper/2505.21549/) (2025)
- [Streaming Drag-Oriented Interactive Video Manipulation: Drag Anything, Anytime!](https://mlsanity.com/paper/2510.03550/) (2025)
- [Hierarchical Semantic Alignment for Image Clustering](https://mlsanity.com/paper/2512.00904/) (2025)
- [Revisiting Robustness for LLM Safety Alignment via Selective Geometry Control](https://mlsanity.com/paper/2602.07340/) (2026)
- [vMFCoOp: Towards Equilibrium on a Unified Hyperspherical Manifold for Prompting Biomedical VLMs](https://mlsanity.com/paper/2511.09540/) (2025)
- [Does CLIP perceive art the same way we do?](https://mlsanity.com/paper/2505.05229/) (2025)
- [Prompt Optimization Meets Subspace Representation Learning for Few-shot Out-of-Distribution Detection](https://mlsanity.com/paper/2509.18111/) (2025)
- [Drift-Augmented Scoring: Text-Derived Noise Robustness for Zero-Shot Audio-Language Classification](https://mlsanity.com/paper/2606.04844/) (2026)

## Abstract (arXiv)

Vision-language models such as CLIP are capable of mapping the different modality data into a unified feature space, enabling zero/few-shot inference by measuring the similarity of given images and texts. However, most existing methods overlook modality gaps in CLIP's encoded features, which is shown as the text and image features lie far apart from each other, resulting in limited classification performance. To tackle this issue, we introduce a method called Selective Vision-Language Subspace Projection (SSP), which incorporates local image features and utilizes them as a bridge to enhance the alignment between image-text pairs. Specifically, our SSP framework comprises two parallel modules: a vision projector and a language projector. Both projectors utilize local image features to span the respective subspaces for image and texts, thereby projecting the image and text features into their respective subspaces to achieve alignment. Moreover, our approach entails only training-free matrix calculations and can be seamlessly integrated into advanced CLIP-based few-shot learning frameworks. Extensive experiments on 11 datasets have demonstrated SSP's superior text-image alignment capabilities, outperforming the state-of-the-art alignment methods. The code is available at https://github.com/zhuhsingyuu/SSP