mlsanityData to 7 Oct 2026

Learning to Decompose Visual Features with Latent Textual Prompts

arXiv 2210.04287 ↗First posted 9 Oct 2022cs.CVcited by 28

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

Recent advances in pre-training vision-language models like CLIP have shown great potential in learning transferable visual representations. Nonetheless, for downstream inference, CLIP-like models suffer from either 1) degraded accuracy and robustness in the case of inaccurate text descriptions during retrieval-based inference (the challenge for zero-shot protocol); or 2) breaking the well-established vision-language alignment (the challenge for linear probing). To address them, we propose Decomposed Feature Prompting (DeFo). DeFo leverages a flexible number of learnable embeddings as textual input while maintaining the vision-language dual-model architecture, which enables the model to learn decomposed visual features with the help of feature-level textual prompts. We further use an additional linear layer to perform classification, allowing a scalable size of language inputs. Our empirical study shows DeFo's significance in improving the vision-language models. For example, DeFo obtains 73.2% test accuracy on ImageNet with a ResNet-50 backbone without tuning any pretrained weights of both the vision and language encoder, outperforming zero-shot CLIP by a large margin of 15.0%, and outperforming state-of-the-art vision-language prompt tuning method by 7.6%.

Key facts. Cited by 28 papers in the map (data through 7 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 30 of its 44 references that are in the map. First posted on arXiv 9 Oct 2022.

Builds on

30 of 44 references.

Closest ancestors · 4

Same direction, newest first

  1. 2022Conditional Prompt Learning for Vision-Language Models10 Mar
  2. 2021Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modeling6 Nov
  3. 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapters9 Oct
  4. 2021Learning to Prompt for Vision-Language Models2 Sep

Other references · 26

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  5. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  6. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  7. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  8. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  9. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  10. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  11. 2013Describing Textures in the Wildcited by 2,110
  12. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,842
14 more
  1. 2013Fine-Grained Visual Classification of Aircraftcited by 1,701
  2. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,507
  3. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,488
  4. 2019Natural Adversarial Examplescited by 1,401
  5. 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,010
  6. 2021StyleCLIP: Text-Driven Manipulation of StyleGAN Imagerycited by 990
  7. 2021CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrievalcited by 645
  8. 2021DenseCLIP: Language-Guided Dense Prediction with Context-Aware Promptingcited by 530
  9. 2020Unsupervised Learning of Dense Visual Representationscited by 147
  10. 2022Language Models with Image Descriptors are Strong Few-Shot Video-Language Learnerscited by 117
  11. 2022CLIP-Event: Connecting Text and Images with Event Structurescited by 86
  12. 2022Learning To Recognize Procedural Activities with Distant Supervisioncited by 85
  13. 2022CP2: Copy-Paste Contrastive Pretraining for Semantic Segmentationcited by 26
  14. 2022Towards Fast Adaptation of Pretrained Contrastive Models for Multi-channel Video-Language Retrievalcited by 16

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 12; 2024: 9; 2025: 6; 2026: 1.

28 papers.

Sort:

By area:

  1. 2023SCLIP: Rethinking Self-Attention for Dense Vision-Language Inferencecited by 169
  2. 2024PromptKD: Unsupervised Prompt Distillation for Vision-Language Modelscited by 72
  3. 2023BadCLIP: Trigger-Aware Prompt Learning for Backdoor Attacks on CLIPcited by 63
  4. 2023DePT: Decoupled Prompt Tuningcited by 58
  5. 2023Improving Zero-Shot Generalization for CLIP with Synthesized Promptscited by 51
  6. 2024Continuous, Subject-Specific Attribute Control in T2I Models by Identifying Semantic Directionscited by 39
  7. 2023Visual Tuningcited by 34
  8. 2024Visual-Text Cross Alignment: Refining the Similarity Score in Vision-Language Modelscited by 30
  9. 2024GalLoP: Learning Global and Local Prompts for Vision-Language Modelscited by 29
  10. 2024ResCLIP: Residual Attention for Training-free Dense Vision-language Inferencecited by 28
  11. 2024AWT: Transferring Vision-Language Models via Augmentation, Weighting, and Transportationcited by 27
  12. 2023Tuning Multi-mode Token-level Prompt Alignment across Modalitiescited by 26
16 more
  1. 2024Concept-Guided Prompt Learning for Generalization in Vision-Language Modelscited by 20
  2. 2023Learning to Adapt CLIP for Few-Shot Monocular Depth Estimationcited by 19
  3. 2023Cross-Modal Concept Learning and Inference for Vision-Language Modelscited by 12
  4. 2023Investigating the Limitation of CLIP Models: The Worst-Performing Categoriescited by 12
  5. 2024Adventurer: Optimizing Vision Mamba Architecture Designs for Efficiencycited by 9
  6. 2025A Closer Look at Conditional Prompt Tuning for Vision-Language Modelscited by 7
  7. 2023BDC-Adapter: Brownian Distance Covariance for Better Vision-Language Reasoningcited by 6
  8. 2023Black-Box Tuning of Vision-Language Models with Effective Gradient Approximationcited by 6
  9. 2025ViT-Linearizer: Distilling Quadratic Knowledge into Linear-Time Vision Modelscited by 5
  10. 2025ViMix-14M: A Curated Multi-Source Video-Text Dataset with Long-Form, High-Quality Captions and Crawl-Free Accesscited by 5
  11. 2025HoPE: Hybrid of Position Embedding for Long Context Vision-Language Modelscited by 4
  12. 2023Prompting classes: Exploring the Power of Prompt Class Learning in Weakly Supervised Semantic Segmentationcited by 3
  13. 2024UMFC: Unsupervised Multi-Domain Feature Calibration for Vision-Language Modelscited by 2
  14. 2025Text to Image for Multi-Label Image Recognition with Joint Prompt-Adapter Learningcited by 1
  15. 2026Let's Roll a BiFTA: Bi-refinement for Fine-grained Text-visual Alignment in Vision-Language Modelscited by 1
  16. 2025LGCA: Enhancing Semantic Representation via Progressive Expansioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023DPL: Decoupled Prompt Learning for Vision-Language Models
  2. 2026What Does Prompt Learning Change? -A Natural-Language Concept Analysis of Vision-Language Models
  3. 2025InPK: Infusing Prior Knowledge into Prompt for Vision-Language Models
  4. 2024Aggregate-and-Adapt Natural Language Prompts for Downstream Generalization of CLIP
  5. 2025Decoupling Augmentation Bias in Prompt Learning for Vision-Language Models
  6. 2023ECO: Ensembling Context Optimization for Vision-Language Models
  7. 2026BiPrompt: Bilateral Prompt Optimization for Visual and Textual Debiasing in Vision-Language Models
  8. 2024Spurious Feature Eraser: Stabilizing Test-Time Adaptation for Vision-Language Foundation Model