mlsanityData to 7 Oct 2026

Task Bias in Vision-Language Models

arXiv 2212.04412 ↗First posted 8 Dec 2022cs.CV cs.LGcited by 8

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

Incidental supervision from language has become a popular approach for learning generic visual representations that can be prompted to perform many recognition tasks in computer vision. We conduct an in-depth exploration of the CLIP model and show that its visual representation is often strongly biased towards solving some tasks more than others. Moreover, which task the representation will be biased towards is unpredictable, with little consistency across images. To resolve this task bias, we show how to learn a visual prompt that guides the representation towards features relevant to their task of interest. Our results show that these visual prompts can be independent of the input image and still effectively provide a conditioning mechanism to steer visual representations towards the desired task.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 30 of its 37 references that are in the map. First posted on arXiv 8 Dec 2022.

Builds on

30 of 37 references.

References · 30

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  6. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  7. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  8. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  9. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  10. 2021Florence: A New Foundation Model for Computer Visioncited by 802
  11. 2020Quantifying Attention Flow in Transformerscited by 759
  12. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 646
18 more
  1. 2021CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrievalcited by 645
  2. 2021CLIPort: What and Where Pathways for Robotic Manipulationcited by 617
  3. 2021Interpretable Machine Learning: Fundamental Principles and 10 Grand Challengescited by 364
  4. 2020Calibrating Deep Neural Networks using Focal Losscited by 353
  5. 2022VQGAN-CLIP: Open Domain Image Generation and Editing with Natural Language Guidancecited by 285
  6. 2022Exploring Visual Prompts for Adapting Large-Scale Modelscited by 271
  7. 2021Simple but Effective: CLIP Embeddings for Embodied AIcited by 191
  8. 2022Patching open-vocabulary models by interpolating weightscited by 187
  9. 2022ZSON: Zero-Shot Object-Goal Navigation using Multimodal Goal Embeddingscited by 177
  10. 2021StyleGAN-NADA: CLIP-Guided Domain Adaptation of Image Generatorscited by 164
  11. 2019Large-scale interactive object segmentation with human annotatorscited by 109
  12. 2022Semantic Abstraction: Open-World 3D Scene Understanding from 2D Vision-Language Modelscited by 95
  13. 2021Uncertainty Baselines: Benchmarks for Uncertainty & Robustness in Deep Learningcited by 81
  14. 2022HyperPrompt: Prompt-based Task-Conditioning of Transformerscited by 67
  15. 2022Disentangling visual and written concepts in CLIPcited by 57
  16. 2021Open Images V5 Text Annotation and Yet Another Mask Text Spottercited by 46
  17. 2020Unadversarial Examples: Designing Objects for Robust Visioncited by 45
  18. 2021True Few-Shot Learning with Prompts -- A Real-World Perspectivecited by 38

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 3; 2025: 1; 2026: 1.

8 papers.

  1. 2024The Neglected Tails in Vision-Language Modelscited by 63
  2. 2023Affective Faces for Goal-Driven Dyadic Communicationcited by 34
  3. 2023Parts of Speech-Grounded Subspaces in Vision-Language Modelscited by 9
  4. 2023Foundation Model-oriented Robustness: Robust Image Model Evaluation with Pretrained Modelscited by 6
  5. 2024Aligning Visual Contrastive learning models via Preference Optimizationcited by 4
  6. 2024S3: Synonymous Semantic Space for Improving Zero-Shot Generalization of Vision-Language Modelscited by 1
  7. 2025Constrained Prompt Enhancement for Improving Zero-Shot Generalization of Vision-Language Modelscited by 1
  8. 2026Language-Instructed Vision Embeddings for Controllable and Generalizable Perceptioncited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2022Prompt Distribution Learning
  2. 2024Learning to Prompt with Text Only Supervision for Vision-Language Models
  3. 2025AmPLe: Supporting Vision-Language Models via Adaptive-Debiased Ensemble Multi-Prompt Learning
  4. 2024Aggregate-and-Adapt Natural Language Prompts for Downstream Generalization of CLIP
  5. 2024Revisiting Prompt Pretraining of Vision-Language Models
  6. 2023Knowledge-Aware Prompt Tuning for Generalizable Vision-Language Models
  7. 2024Fully Fine-tuned CLIP Models are Efficient Few-Shot Learners
  8. 2023Understanding the Multi-modal Prompts of the Pre-trained Vision-Language Model