mlsanityData to 7 Oct 2026

Multimodal Explanations: Justifying Decisions and Pointing to the Evidence

arXiv 1802.08129 ↗First posted 15 Feb 2018cs.AI cs.CL cs.CVcited by 179

Not on the map. Why?

Abstract (arXiv)

Deep models that are both effective and explainable are desirable in many settings; prior explainable models have been unimodal, offering either image-based visualization of attention weights or text-based generation of post-hoc justifications. We propose a multimodal approach to explanation, and argue that the two modalities provide complementary explanatory strengths. We collect two new datasets to define and evaluate this task, and propose a novel model which can provide joint textual rationale generation and attention visualization. Our datasets define visual and textual justifications of a classification decision for activity recognition tasks (ACT-X) and for visual question answering tasks (VQA-X). We quantitatively show that training with the textual explanations not only yields better textual justification models, but also better localizes the evidence that supports the decision. We also qualitatively show cases where visual explanation is more insightful than textual explanation, and vice versa, supporting our thesis that multimodal explanation models offer significant benefits over unimodal approaches.

Key facts. Cited by 179 papers in the map (data through 7 Oct 2026). First posted on arXiv 15 Feb 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 35; 2022: 35; 2023: 43; 2024: 26; 2025: 25; 2026: 15.

179 papers.

Sort:

By area:

  1. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 702
  2. 2023ViperGPT: Visual Inference via Python Execution for Reasoningcited by 591
  3. 2021Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Surveycited by 457
  4. 2022Visual Classification via Description from Large Language Modelscited by 331
  5. 2022Language in a Bottle: Language Model Guided Concept Bottlenecks for Interpretable Image Classificationcited by 310
  6. 2022From Anecdotal Evidence to Quantitative Evaluation Methods: A Systematic Review on Evaluating Explainable AIcited by 299
  7. 2021An Empirical Study of GPT-3 for Few-Shot Knowledge-Based VQAcited by 298
  8. 2022Natural Language Descriptions of Deep Visual Featurescited by 136
  9. 2022Towards Faithful Model Explanation in NLP: A Surveycited by 129
  10. 2021Teach Me to Explain: A Review of Datasets for Explainable Natural Language Processingcited by 126
  11. 2021Explainable AI (XAI): A Systematic Meta-Survey of Current Challenges and Future Opportunitiescited by 125
  12. 2021Reframing Human-AI Collaboration for Generating Free-Text Explanationscited by 125
38 more
  1. 2022Foundations and Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questionscited by 116
  2. 2022Language Models are General-Purpose Interfacescited by 96
  3. 2021Medical Visual Question Answering: A Surveycited by 93
  4. 2023Do Models Explain Themselves? Counterfactual Simulatability of Natural Language Explanationscited by 85
  5. 2021e-ViL: A Dataset and Benchmark for Natural Language Explanations in Vision-Language Taskscited by 84
  6. 2021Explainability of deep vision-based autonomous driving systems: Review and challengescited by 80
  7. 2022Diffusion Models for Counterfactual Explanationscited by 79
  8. 2023Explaining black box text modules in natural language with language modelscited by 61
  9. 2022NLX-GPT: A Model for Natural Language Explanations in Vision and Vision-Language Taskscited by 60
  10. 2025Grounded Chain-of-Thought for Multimodal Large Language Modelscited by 57
  11. 2022Grounding Answers for Visual Questions Asked by Visually Impaired Peoplecited by 47
  12. 2021A Review on Explainability in Multimodal Deep Neural Netscited by 46
  13. 2021Diagnosing Vision-and-Language Navigation: What Really Matterscited by 36
  14. 2022MultiViz: Towards Visualizing and Understanding Multimodal Modelscited by 36
  15. 2022Going Beyond XAI: A Systematic Survey for Explanation-Guided Learningcited by 35
  16. 2025Visual Large Language Models for Generalized and Specialized Applicationscited by 35
  17. 2021A Study of Automatic Metrics for the Evaluation of Natural Language Explanationscited by 34
  18. 2021Greedy Gradient Ensemble for Robust Visual Question Answeringcited by 34
  19. 2022DIME: Fine-grained Interpretations of Multimodal Models via Disentangled Local Explanationscited by 34
  20. 2025A Survey on Mechanistic Interpretability for Multi-Modal Foundation Modelscited by 34
  21. 2021Select, Substitute, Search: A New Benchmark for Knowledge-Augmented Visual Question Answeringcited by 33
  22. 2021Local Interpretations for Explainable Natural Language Processing: A Surveycited by 31
  23. 2021On the Diversity and Limits of Human Explanationscited by 30
  24. 2023Beyond Task Performance: Evaluating and Reducing the Flaws of Large Multimodal Models with In-Context Learningcited by 30
  25. 2025PointArena: Probing Multimodal Grounding Through Language-Guided Pointingcited by 30
  26. 2025I2MoE: Interpretable Multimodal Interaction-aware Mixture-of-Expertscited by 29
  27. 2021Towards a Collective Agenda on AI for Earth Science Data Analysiscited by 27
  28. 2022Explaining Chest X-ray Pathologies in Natural Languagecited by 27
  29. 2025Explain Before You Answer: A Survey on Compositional Visual Reasoningcited by 26
  30. 2023Multimodal Explainable Artificial Intelligence: A Comprehensive Review of Methodological Advances and Future Research Directionscited by 25
  31. 2021Knowledge-Grounded Self-Rationalization via Extractive and Natural Language Explanationscited by 24
  32. 2021Separating Skills and Concepts for Novel Visual Question Answeringcited by 24
  33. 2024Properties and Challenges of LLM-Generated Explanationscited by 24
  34. 2021A Joint Network for Grasp Detection Conditioned on Natural Language Commandscited by 23
  35. 2023UNK-VQA: A Dataset and a Probe into the Abstention Ability of Multi-modal Large Modelscited by 23
  36. 2024Explainable Generative AI (GenXAI): A Survey, Conceptualization, and Research Agendacited by 22
  37. 2021Levels of explainable artificial intelligence for human-aligned conversational explanationscited by 21
  38. 2023Discovering Spatio-Temporal Rationales for Video Question Answeringcited by 21

50 most cited of 179 All 179 (JSON)