# TextRefiner: Internal Visual Feature as Efficient Refiner for Vision-Language Models Prompt Tuning

arXiv: https://arxiv.org/abs/2412.08176 · HTML: https://mlsanity.com/paper/2412.08176/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 4 of its 51 references that are in the map.
- First posted on arXiv 11 Dec 2024.

## Placed in

- Multimodal LLMs / vision-language models / [Prompt learning and adaptation of CLIP](https://mlsanity.com/mllm/prompt-learning-and-adaptation-of-clip/)

## Builds on (most cited first, 4 of 4)

- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Fine-Grained Visual Classification of Aircraft](https://mlsanity.com/paper/1306.5151/) (2013)
- [What do Vision Transformers Learn? A Visual Exploration](https://mlsanity.com/paper/2212.06727/) (2022)

## Built on by (most cited first, 13 of 13)

- [LLM-empowered Dynamic Prompt Routing for Vision-Language Models Tuning under Long-Tailed Distributions](https://mlsanity.com/paper/2508.15688/) (2025)
- [Beyond Graph Model: Reliable VLM Fine-Tuning via Random Graph Adapter](https://mlsanity.com/paper/2507.10355/) (2025)
- [Spotlighter: Revisiting Prompt Tuning from a Representative Mining View](https://mlsanity.com/paper/2509.00905/) (2025)
- [AnchorOPT: Towards Optimizing Dynamic Anchors for Adaptive Prompt Learning](https://mlsanity.com/paper/2511.21188/) (2025)
- [Prompt Tuning for CLIP on the Pretrained Manifold](https://mlsanity.com/paper/2602.19198/) (2026)
- [DeAR: Fine-Grained VLM Adaptation by Decomposing Attention Head Roles](https://mlsanity.com/paper/2603.01111/) (2026)
- [NEARL: Interacted Query Adaptation with Orthogonal Regularization for Medical Vision-Language Understanding](https://mlsanity.com/paper/2508.04101/) (2025)
- [Cross-Domain Few-Shot Learning via Multi-View Collaborative Optimization with Vision-Language Models](https://mlsanity.com/paper/2508.12861/) (2025)
- [Preserve and Sculpt: Manifold-Aligned Fine-tuning of Vision-Language Models for Few-Shot Learning](https://mlsanity.com/paper/2508.12877/) (2025)
- [GA2-CLIP: Generic Attribute Anchor for Efficient Prompt Tuningin Video-Language Models](https://mlsanity.com/paper/2511.22125/) (2025)
- [CAPT: Confusion-Aware Prompt Tuning for Reducing Vision-Language Misalignment](https://mlsanity.com/paper/2603.02557/) (2026)
- [MQAdapter: Multi-Modal Quantum Adapter for Coarse-to-Fine VLM Fine-tuning](https://mlsanity.com/paper/2607.12418/) (2026)
- [What Does Prompt Learning Change? -A Natural-Language Concept Analysis of Vision-Language Models](https://mlsanity.com/paper/2608.24142/) (2026)

## Abstract (arXiv)

Despite the efficiency of prompt learning in transferring vision-language models (VLMs) to downstream tasks, existing methods mainly learn the prompts in a coarse-grained manner where the learned prompt vectors are shared across all categories. Consequently, the tailored prompts often fail to discern class-specific visual concepts, thereby hindering the transferred performance for classes that share similar or complex visual attributes. Recent advances mitigate this challenge by leveraging external knowledge from Large Language Models (LLMs) to furnish class descriptions, yet incurring notable inference costs. In this paper, we introduce TextRefiner, a plug-and-play method to refine the text prompts of existing methods by leveraging the internal knowledge of VLMs. Particularly, TextRefiner builds a novel local cache module to encapsulate fine-grained visual concepts derivedfrom local tokens within the image branch. By aggregating and aligning the cached visual descriptions with the original output of the text branch, TextRefiner can efficiently refine and enrich the learned prompts from existing methods without relying on any external expertise. For example, it improves the performance of CoOp from 71.66 % to 76.94 % on 11 benchmarks, surpassing CoCoOp which introduces instance-wise features for text prompts. Equipped with TextRefiner, PromptKD achieves state-of-the-art performance and is efficient in inference. Our code is relesed at https://github.com/xjjxmu/TextRefiner