# SEP: Self-Enhanced Prompt Tuning for Visual-Language Model

arXiv: https://arxiv.org/abs/2405.15549 · HTML: https://mlsanity.com/paper/2405.15549/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 3 of its 54 references that are in the map.
- First posted on arXiv 24 May 2024.

## Placed in

- Multimodal LLMs / vision-language models / [Prompt learning and adaptation of CLIP](https://mlsanity.com/mllm/prompt-learning-and-adaptation-of-clip/)

## Builds on (most cited first, 3 of 3)

- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairs](https://mlsanity.com/paper/2111.02114/) (2021)
- [Gradient-Regulated Meta-Prompt Learning for Generalizable Vision-Language Models](https://mlsanity.com/paper/2303.06571/) (2023)

## Built on by (most cited first, 3 of 3)

- [Rethinking Few Shot CLIP Benchmarks: A Critical Analysis in the Inductive Setting](https://mlsanity.com/paper/2507.20834/) (2025)
- [Chain-of-Look Spatial Reasoning for Dense Surgical Instrument Counting](https://mlsanity.com/paper/2602.11024/) (2026)
- [Integrated Structural Prompt Learning for Vision-Language Models](https://mlsanity.com/paper/2507.05677/) (2025)

## Abstract (arXiv)

Prompt tuning based on Context Optimization (CoOp) effectively adapts visual-language models (VLMs) to downstream tasks by inferring additional learnable prompt tokens. However, these tokens are less discriminative as they are independent of the pre-trained tokens and fail to capture input-specific knowledge, such as class-aware textual or instance-aware visual knowledge. Leveraging the discriminative and generalization capabilities inherent in pre-trained tokens, we introduce a novel approach named Self-Enhanced Prompt Tuning (SEP). The core principle of SEP involves adapting the learnable prompt tokens at each encoder layer from the corresponding self-pretrained tokens, thereby explicitly incorporating discriminative prior knowledge to enhance both textual-level and visual-level embeddings. Furthermore, SEP's self-enhanced tokens not only boost discrimination but also mitigate domain shifts in unseen domains, enhancing generalization. In practice, SEP selects several representative tokens from all pre-trained tokens for each input data at every layer of the text/visual encoders. Subsequently, a Token Fusion Module (TFM) is introduced to generate a self-enhanced token by merging these representative tokens with the learnable tokens using a cross-attention mechanism. This self-enhanced token is then concatenated with all pre-trained tokens, serving as input for subsequent encoder layers to produce the relevant embeddings. Comprehensive evaluations across various benchmarks and tasks confirm SEP's efficacy in prompt tuning. Code: hrefCodehttps://github.com/htyao89/SEP.