mlsanityData to 7 Oct 2026

PerceptionCLIP: Visual Classification by Inferring and Conditioning on Contexts

arXiv 2308.01313 ↗First posted 2 Aug 2023cs.CV cs.AI cs.CLcited by 16

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

Vision-language models like CLIP are widely used in zero-shot image classification due to their ability to understand various visual concepts and natural language descriptions. However, how to fully leverage CLIP's unprecedented human-like understanding capabilities to achieve better performance is still an open question. This paper draws inspiration from the human visual perception process: when classifying an object, humans first infer contextual attributes (e.g., background and orientation) which help separate the foreground object from the background, and then classify the object based on this information. Inspired by it, we observe that providing CLIP with contextual attributes improves zero-shot image classification and mitigates reliance on spurious features. We also observe that CLIP itself can reasonably infer the attributes from an image. With these observations, we propose a training-free, two-step zero-shot classification method PerceptionCLIP. Given an image, it first infers contextual attributes (e.g., background) and then performs object classification conditioning on them. Our experiments show that PerceptionCLIP achieves better generalization, group robustness, and interoperability. Our code is available at https://github.com/umd-huang-lab/perceptionCLIP

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 41 of its 56 references that are in the map. First posted on arXiv 2 Aug 2023.

Builds on

41 of 56 references.

Closest ancestors · 10

Same direction, newest first

  1. 2023Waffling around for Performance: Visual Classification with Random Words and Broad Concepts12 Jun
  2. 2023Enhancing CLIP with CLIP: Exploring Pseudolabeling for Limited-Label Prompt Tuning2 Jun
  3. 2022SuS-X: Training-Free Name-Only Transfer of Vision-Language Models28 Nov
  4. 2022Visual Classification via Description from Large Language Models13 Oct
  5. 2022When and why vision-language models behave like bags-of-words, and what to do about it?4 Oct
  6. 2022Test-Time Prompt Tuning for Zero-Shot Generalization in Vision-Language Models15 Sep
  7. 2022Prompt-aligned Gradient for Prompt Tuning30 May
  8. 2022Unsupervised Prompt Learning for Vision-Language Models7 Apr
  9. 2022Conditional Prompt Learning for Vision-Language Models10 Mar
  10. 2021Learning to Prompt for Vision-Language Models2 Sep

Other references · 31

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2023Segment Anythingcited by 8,158
  5. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,690
  6. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  7. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  8. 2014Deep Learning Face Attributes in the Wildcited by 3,679
  9. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  10. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  11. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  12. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,842
19 more
  1. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  2. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,507
  3. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,488
  4. 2019Natural Adversarial Examplescited by 1,401
  5. 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,010
  6. 2022MedCLIP: Contrastive Learning from Unpaired Medical Images and Textcited by 564
  7. 2021Just Train Twice: Improving Group Robustness without Training Group Informationcited by 522
  8. 2022What does a platypus look like? Generating customized prompts for zero-shot image classificationcited by 263
  9. 2023Improving CLIP Training with Language Rewritescited by 216
  10. 2023Debiasing Vision-Language Models via Biased Promptscited by 147
  11. 2021Evaluating CLIP: Towards Characterization of Broader Capabilities and Downstream Implicationscited by 128
  12. 2018Benchmarking Neural Network Robustness to Common Corruptions and Surface Variationscited by 99
  13. 2023CHiLS: Zero-Shot Image Classification with Hierarchical Label Setscited by 79
  14. 2023GeoDE: a Geographically Diverse Evaluation Dataset for Object Recognitioncited by 50
  15. 2023Mitigating Spurious Correlations in Multi-modal Models during Fine-tuningcited by 47
  16. 2023LaFTer: Label-Free Tuning of Zero-shot Classifier using Language and Unlabeled Image Collectionscited by 43
  17. 2022Doubly Right Object Recognition: A Why Prompt for Visual Rationalescited by 29
  18. 2023CLIPER: A Unified Vision-Language Framework for In-the-Wild Facial Expression Recognitioncited by 27
  19. 2023DesCo: Learning Object Recognition with Rich Language Descriptionscited by 25

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 5; 2025: 4; 2026: 6.

16 papers.

Sort:

By area:

  1. 2025Pixels Versus Priors: Controlling Knowledge Priors in Vision-Language Models through Visual Counterfactscited by 30
  2. 2024DPA: Dual Prototypes Alignment for Unsupervised Adaptation of Vision-Language Modelscited by 5
  3. 2024A sound description: Exploring prompt templates and class descriptions to enhance zero-shot audio classificationcited by 3
  4. 2025Order Matters: On Parameter-Efficient Image-to-Video Probing for Recognizing Nearly Symmetric Actionscited by 3
  5. 2024ContextHOI: Spatial Context Learning for Human-Object Interaction Detectioncited by 2
  6. 2025Tile-Based ViT Inference with Visual-Cluster Priors for Zero-Shot Multi-Species Plant Identificationcited by 2
  7. 2025Representation-Level Counterfactual Calibration for Debiased Zero-Shot Recognitioncited by 2
  8. 2024CoAPT: Context Attribute words for Prompt Tuningcited by 1
  9. 2023Noise-Tolerant Few-Shot Unsupervised Adapter for Vision-Language Modelscited by 0
  10. 2024DoubleCCA: Improving Foundation Model Group Robustness with Random Sentence Embeddingscited by 0
  11. 2026Density-Aware Translation of Spurious Correlations in Zero-Shot VLMscited by 0
  12. 2026SADL: What to Ignore? A Benchmark for Subject-Aware Distractor Localizationcited by 0
4 more
  1. 2026ProCal: Inference-Time Proposal Calibration for Open-Vocabulary Object Detectioncited by 0
  2. 2026Fairness-Aware Test-Time Prompt Tuningcited by 0
  3. 2026Re-calibrated Contrastive Loss for Transformation-Aware Prompt Conditioning in Vision-Language Modelscited by 0
  4. 2026Combining Hierarchical Cognitive Process with Process Supervision for Interpretable Scene Safety Understandingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Real Classification by Description: Extending CLIP's Limits of Part Attributes Recognition
  2. 2026Explaining CLIP Zero-shot Predictions Through Concepts
  3. 2023AutoCLIP: Auto-tuning Zero-Shot Classifiers for Vision-Language Models
  4. 2025DeCLIP: Decoupled Learning for Open-Vocabulary Dense Perception
  5. 2023ECO: Ensembling Context Optimization for Vision-Language Models
  6. 2022Tip-Adapter: Training-free Adaption of CLIP for Few-shot Classification
  7. 2023Zero-Shot Visual Classification with Guided Cropping
  8. 2023What does CLIP know about a red circle? Visual prompt engineering for VLMs