mlsanityData to 7 Oct 2026

Position: Do Not Explain Vision Models Without Context

arXiv 2404.18316 ↗First posted 28 Apr 2024cs.CV cs.AI cs.LGcited by 2

Not on the map. Why?

Abstract (arXiv)

Does the stethoscope in the picture make the adjacent person a doctor or a patient? This, of course, depends on the contextual relationship of the two objects. If it's obvious, why don't explanation methods for vision models use contextual information? In this paper, we (1) review the most popular methods of explaining computer vision models by pointing out that they do not take into account context information, (2) show examples of failures of popular XAI methods, (3) provide examples of real-world use cases where spatial context plays a significant role, (4) propose new research directions that may lead to better use of context information in explaining computer vision models, (5) argue that a change in approach to explanations is needed from 'where' to 'how'.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 35 of its 57 references that are in the map. First posted on arXiv 28 Apr 2024.

Builds on

35 of 57 references.

References · 35

  1. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  2. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  3. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,690
  4. 2017A Unified Approach to Interpreting Model Predictionscited by 6,488
  5. 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,040
  6. 2016Feature Pyramid Networks for Object Detectioncited by 5,021
  7. 2013Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Mapscited by 2,348
  8. 2013Visualizing and Understanding Convolutional Networkscited by 2,278
  9. 2022Visual Prompt Tuningcited by 1,674
  10. 2021MLP-Mixer: An all-MLP Architecture for Visioncited by 1,441
  11. 2016Context Encoders: Feature Learning by Inpaintingcited by 1,424
  12. 2015Unsupervised Visual Representation Learning by Context Predictioncited by 1,062
23 more
  1. 2020Data Efficient and Weakly Supervised Computational Pathology on Whole Slide Imagescited by 738
  2. 2017Dynamic Routing Between Capsulescited by 689
  3. 2023ViperGPT: Visual Inference via Python Execution for Reasoningcited by 591
  4. 2021TransMIL: Transformer based Correlated Multiple Instance Learning for Whole Slide Image Classificationcited by 549
  5. 2022Visual Programming: Compositional visual reasoning without trainingcited by 507
  6. 2022Scaling Vision Transformers to Gigapixel Images via Hierarchical Self-Supervised Learningcited by 357
  7. 2022DALL-Eval: Probing the Reasoning Skills and Social Biases of Text-to-Image Generation Modelscited by 230
  8. 2016Fine-grained Analysis of Sentence Embeddings Using Auxiliary Prediction Taskscited by 216
  9. 2023Holistic Evaluation of Text-To-Image Modelscited by 160
  10. 2021Explainable artificial intelligence (XAI) in deep learning-based medical image analysiscited by 147
  11. 2022From Attribution Maps to Human-Understandable Explanations through Concept Relevance Propagationcited by 141
  12. 2023COLA: A Benchmark for Compositional Text-to-image Retrievalcited by 53
  13. 2022A Review of Emerging Research Directions in Abstract Visual Reasoningcited by 33
  14. 2023Interpretability Benchmark for Evaluating Spatial Misalignment of Prototypical Parts Explanationscited by 27
  15. 2016A Deep Spatial Contextual Long-term Recurrent Convolutional Network for Saliency Detectioncited by 23
  16. 2019Enriching Visual with Verbal Explanations for Relational Concepts -- Combining LIME with Alephcited by 23
  17. 2023Context Understanding in Computer Vision: A Surveycited by 23
  18. 2021ProtoMIL: Multiple Instance Learning with Prototypical Parts for Whole-Slide Image Classificationcited by 22
  19. 2017Identifying Spatial Relations in Images using Convolutional Neural Networkscited by 4
  20. 2021Spatial-context-aware deep neural network for multi-class image classificationcited by 4
  21. 2020End-to-End Vision-Based Adaptive Cruise Control (ACC) Using Deep Reinforcement Learningcited by 1
  22. 2021A Feature Fusion-Net Using Deep Spatial Context Encoder and Nonstationary Joint Statistical Model for High Resolution SAR Image Classificationcited by 1
  23. 2024Deep spatial context: when attention-based models meet spatial regressioncited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Computational Hermeneutics: Evaluating generative AI as a cultural technologycited by 7
  2. 2026We Need Explanation Cards to Connect Explanation Algorithms to the Real Worldcited by 0