mlsanityData to 7 Oct 2026

Doubly Right Object Recognition: A Why Prompt for Visual Rationales

arXiv 2212.06202 ↗First posted 12 Dec 2022cs.CVcited by 29

Not on the map. Why?

Abstract (arXiv)

Many visual recognition models are evaluated only on their classification accuracy, a metric for which they obtain strong performance. In this paper, we investigate whether computer vision models can also provide correct rationales for their predictions. We propose a ``doubly right'' object recognition benchmark, where the metric requires the model to simultaneously produce both the right labels as well as the right rationales. We find that state-of-the-art visual models, such as CLIP, often provide incorrect rationales for their categorical predictions. However, by transferring the rationales from language models into visual representations through a tailored dataset, we show that we can learn a ``why prompt,'' which adapts large visual representations to produce correct rationales. Visualizations and empirical experiments show that our prompts significantly improve performance on doubly right object recognition, in addition to zero-shot transfer to unseen tasks and datasets.

Key facts. Cited by 29 papers in the map (data through 7 Oct 2026). Builds on 45 of its 66 references that are in the map. First posted on arXiv 12 Dec 2022.

Builds on

45 of 66 references.

References · 45

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  6. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  7. 2017A Unified Approach to Interpreting Model Predictionscited by 6,488
  8. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  9. 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,040
  10. 2015VQA: Visual Question Answeringcited by 2,489
  11. 2013Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Mapscited by 2,348
  12. 2013Visualizing and Understanding Convolutional Networkscited by 2,278
33 more
  1. 2021Learning to Prompt for Vision-Language Modelscited by 2,269
  2. 2019Language Models as Knowledge Bases?cited by 1,922
  3. 2022Visual Prompt Tuningcited by 1,674
  4. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,530
  5. 2017Learning Important Features Through Propagating Activation Differencescited by 1,454
  6. 2022Conditional Prompt Learning for Vision-Language Modelscited by 1,427
  7. 2017SmoothGrad: removing noise by adding noisecited by 1,097
  8. 2017Network Dissection: Quantifying Interpretability of Deep Visual Representationscited by 703
  9. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 646
  10. 2018From Recognition to Cognition: Visual Commonsense Reasoningcited by 553
  11. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
  12. 2022Visual Classification via Description from Large Language Modelscited by 331
  13. 2021Generic Attention-model Explainability for Interpreting Bi-Modal and Encoder-Decoder Transformerscited by 316
  14. 2022Exploring Visual Prompts for Adapting Large-Scale Modelscited by 271
  15. 2017Zero-Shot Learning -- The Good, the Bad and the Uglycited by 254
  16. 2022Visual Prompting via Image Inpaintingcited by 248
  17. 2018Textual Explanations for Self-Driving Vehiclescited by 226
  18. 2016Synthesizing the preferred inputs for neurons in neural networks via deep generator networkscited by 181
  19. 2016Generating Visual Explanationscited by 156
  20. 2021Salient ImageNet: How to discover spurious features in Deep Learning?cited by 114
  21. 2020Explainable deep learning models in medical image analysiscited by 111
  22. 2021Learning to Predict Visual Attributes in the Wildcited by 95
  23. 2022K-LITE: Learning Transferable Visual Models with External Knowledgecited by 79
  24. 2017Learning to Act Properly: Predicting and Explaining Affordances from Imagescited by 74
  25. 2017MovieGraphs: Towards Understanding Human-Centric Situations from Videoscited by 67
  26. 2022Prompt Consistency for Zero-Shot Task Generalizationcited by 66
  27. 2019The iMaterialist Fashion Attribute Datasetcited by 37
  28. 2022Fine-tuning Image Transformers using Learnable Memorycited by 34
  29. 2019Testing the robustness of attribution methods for convolutional neural networks in MRI-based Alzheimer's disease classificationcited by 25
  30. 2016"What happens if..." Learning to Predict the Effect of Forces in Imagescited by 23
  31. 2022Causal Transportability for Visual Recognitioncited by 23
  32. 2018Interpretable Intuitive Physics Modelcited by 19
  33. 2019Knowledge-Augmented Language Model and its Application to Unsupervised Named-Entity Recognitioncited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 11; 2024: 12; 2025: 3; 2026: 3.

29 papers.

Sort:

By area:

  1. 2023ViperGPT: Visual Inference via Python Execution for Reasoningcited by 591
  2. 2023Waffling around for Performance: Visual Classification with Random Words and Broad Conceptscited by 100
  3. 2023Learning without Forgetting for Vision-Language Modelscited by 79
  4. 2024SelfIE: Self-Interpretation of Large Language Model Embeddingscited by 63
  5. 2024Follow the Rules: Reasoning for Video Anomaly Detection with Large Language Modelscited by 55
  6. 2023CoTDet: Affordance Knowledge Prompting for Task Driven Object Detectioncited by 36
  7. 2024Convolutional Prompting meets Language Models for Continual Learningcited by 32
  8. 2024Facing the Elephant in the Room: Visual Prompt Tuning or Full Finetuning?cited by 31
  9. 2023AttrSeg: Open-Vocabulary Semantic Segmentation via Attribute Decomposition-Aggregationcited by 29
  10. 2024Do Vision-Language Models Really Understand Visual Language?cited by 29
  11. 2023Convolutional Visual Prompt for Robust Visual Perceptioncited by 17
  12. 2023PerceptionCLIP: Visual Classification by Inferring and Conditioning on Contextscited by 16
17 more
  1. 2023Object Recognition as Next Token Predictioncited by 12
  2. 2023Interpreting and Controlling Vision Foundation Models via Text Explanationscited by 11
  3. 2025Re-Imagining Multimodal Instruction Tuning: A Representation Viewcited by 9
  4. 2024StimuVAR: Spatiotemporal Stimuli-aware Video Affective Reasoning with Multimodal Large Language Modelscited by 8
  5. 20233VL: Using Trees to Improve Vision-Language Models' Interpretabilitycited by 7
  6. 2024Tree of Attributes Prompt Learning for Vision-Language Modelscited by 7
  7. 2024ECOR: Explainable CLIP for Object Recognitioncited by 4
  8. 2024Language Model as Visual Explainercited by 3
  9. 2024Cross-Modal Attention Alignment Network with Auxiliary Text Description for zero-shot sketch-based image retrievalcited by 2
  10. 2024Beyond Accuracy: On the Effects of Fine-tuning Towards Vision-Language Model's Prediction Rationalitycited by 2
  11. 2025Multi-Rationale Explainable Object Recognition via Contrastive Conditional Inferencecited by 1
  12. 2026Language-Instructed Vision Embeddings for Controllable and Generalizable Perceptioncited by 1
  13. 2023Leveraging Vision-Language Foundation Models for Fine-Grained Downstream Taskscited by 0
  14. 2024ExIQA: Explainable Image Quality Assessment Using Distortion Attributescited by 0
  15. 2025Dynamic Multi-level Weighted Alignment Network for Zero-shot Sketch-based Image Retrievalcited by 0
  16. 2026Toward Low-Latency Vision-Language Models with Doubly-Correct Predictions in Egocentric Visual Understandingcited by 0
  17. 2026ReCalMatch:Reliability-Calibrated Semantic Guidance for Semi-Supervised Fine-Grained Recognitioncited by 0