mlsanityData to 7 Oct 2026

Generating Visual Explanations

arXiv 1603.08507 ↗First posted 28 Mar 2016cs.CV cs.AI cs.CLcited by 156

Not on the map. Why?

Abstract (arXiv)

Clearly explaining a rationale for a classification decision to an end-user can be as important as the decision itself. Existing approaches for deep visual recognition are generally opaque and do not output any justification text; contemporary vision-language models can describe image content but fail to take into account class-discriminative image aspects which justify visual predictions. We propose a new model that focuses on the discriminating properties of the visible object, jointly predicts a class label, and explains why the predicted label is appropriate for the image. We propose a novel loss function based on sampling and reinforcement learning that learns to generate sentences that realize a global sentence property, such as class specificity. Our results on a fine-grained bird species classification dataset show that our model is able to generate explanations which are not only consistent with an image but also more discriminative than descriptions produced by existing captioning methods.

Key facts. Cited by 156 papers in the map (data through 7 Oct 2026). First posted on arXiv 28 Mar 2016.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 44; 2022: 33; 2023: 35; 2024: 26; 2025: 12; 2026: 6.

156 papers.

Sort:

By area:

  1. 2022Language in a Bottle: Language Model Guided Concept Bottlenecks for Interpretable Image Classificationcited by 310
  2. 2022Can language models learn from explanations in context?cited by 248
  3. 2023Symbolic Chain-of-Thought Distillation: Small Models Can Also "Think" Step-by-Stepcited by 144
  4. 2021From Show to Tell: A Survey on Deep Learning-based Image Captioningcited by 137
  5. 2022Natural Language Descriptions of Deep Visual Featurescited by 136
  6. 2022Towards Faithful Model Explanation in NLP: A Surveycited by 129
  7. 2022Toward Transparent AI: A Survey on Interpreting the Inner Structures of Deep Neural Networkscited by 126
  8. 2021Reframing Human-AI Collaboration for Generating Free-Text Explanationscited by 125
  9. 2022Explanations from Large Language Models Make Small Reasoners Bettercited by 123
  10. 2024Rethinking Interpretability in the Era of Large Language Modelscited by 110
  11. 2022Diffusion Visual Counterfactual Explanationscited by 96
  12. 2024Discover-then-Name: Task-Agnostic Concept Bottlenecks via Automated Concept Discoverycited by 91
38 more
  1. 2021A Comprehensive Taxonomy for Explainable Artificial Intelligence: A Systematic Survey of Surveys on Methods and Conceptscited by 90
  2. 2021The effectiveness of feature attribution methods and its correlation with automatic evaluation scorescited by 87
  3. 2021e-ViL: A Dataset and Benchmark for Natural Language Explanations in Vision-Language Taskscited by 84
  4. 2021Few-Shot Self-Rationalization with Natural Language Promptscited by 83
  5. 2021Explainability of deep vision-based autonomous driving systems: Review and challengescited by 80
  6. 2022Diffusion Models for Counterfactual Explanationscited by 79
  7. 2021Explainable Artificial Intelligence for Autonomous Driving: A Comprehensive Overview and Field Guide for Future Research Directionscited by 73
  8. 2024SelfIE: Self-Interpretation of Large Language Model Embeddingscited by 63
  9. 2023Explaining black box text modules in natural language with language modelscited by 61
  10. 2022NLX-GPT: A Model for Natural Language Explanations in Vision and Vision-Language Taskscited by 60
  11. 2021D3Net: A Unified Speaker-Listener Architecture for 3D Dense Captioning and Visual Groundingcited by 52
  12. 2022A Survey on Legal Judgment Prediction: Datasets, Metrics, Models and Challengescited by 50
  13. 2023FunnyBirds: A Synthetic Vision Dataset for a Part-Based Analysis of Explainable AI Methodscited by 49
  14. 2021A Review on Explainability in Multimodal Deep Neural Netscited by 46
  15. 2023Rigorously Assessing Natural Language Explanations of Neuronscited by 44
  16. 2024Incremental Residual Concept Bottleneck Modelscited by 43
  17. 2021Diagnosing Vision-and-Language Navigation: What Really Matterscited by 36
  18. 2022Augmenting Interpretable Models with LLMs during Trainingcited by 36
  19. 2023Human-Centric Multimodal Machine Learning: Recent Advances and Testbed on AI-based Recruitmentcited by 32
  20. 2023FIND: A Function Description Benchmark for Evaluating Interpretability Methodscited by 32
  21. 2021Local Interpretations for Explainable Natural Language Processing: A Surveycited by 31
  22. 2022Sparse Visual Counterfactual Explanations in Image Spacecited by 30
  23. 2022Grounding Visual Representations with Texts for Domain Generalizationcited by 30
  24. 2023GaitGCI: Generative Counterfactual Intervention for Gait Recognitioncited by 30
  25. 2022Doubly Right Object Recognition: A Why Prompt for Visual Rationalescited by 29
  26. 2021Explain, Edit, and Understand: Rethinking User Study Design for Evaluating Model Explanationscited by 26
  27. 2021Principles of Explanation in Human-AI Systemscited by 25
  28. 2021Revisiting The Evaluation of Class Activation Mapping for Explainability: A Novel Metric and Experimental Analysiscited by 25
  29. 2021Nice perfume. How long did you marinate in it? Multimodal Sarcasm Explanationcited by 25
  30. 2023Multimodal Explainable Artificial Intelligence: A Comprehensive Review of Methodological Advances and Future Research Directionscited by 25
  31. 2021Knowledge-Grounded Self-Rationalization via Extractive and Natural Language Explanationscited by 24
  32. 2021Measuring and Improving BERT's Mathematical Abilities by Predicting the Order of Reasoningcited by 23
  33. 2022Listen to Interpret: Post-hoc Interpretability for Audio Networks with NMFcited by 23
  34. 2021Levels of explainable artificial intelligence for human-aligned conversational explanationscited by 21
  35. 2021Connecting Attributions and QA Model Behavior on Realistic Counterfactualscited by 20
  36. 2021CX-ToM: Counterfactual Explanations with Theory-of-Mind for Enhancing Human Trust in Image Recognition Modelscited by 20
  37. 2023Semantic HELM: A Human-Readable Memory for Reinforcement Learningcited by 17
  38. 2021Contextualized Keyword Representations for Multi-modal Retinal Image Captioningcited by 16

50 most cited of 156 All 156 (JSON)