mlsanityData to 7 Oct 2026

Faithful Multimodal Explanation for Visual Question Answering

arXiv 1809.02805 ↗First posted 8 Sep 2018cs.CL cs.CVcited by 64

Not on the map. Why?

Abstract (arXiv)

AI systems' ability to explain their reasoning is critical to their utility and trustworthiness. Deep neural networks have enabled significant progress on many challenging problems such as visual question answering (VQA). However, most of them are opaque black boxes with limited explanatory capability. This paper presents a novel approach to developing a high-performing VQA system that can elucidate its answers with integrated textual and visual explanations that faithfully reflect important aspects of its underlying reasoning while capturing the style of comprehensible human explanations. Extensive experimental evaluation demonstrates the advantages of this approach compared to competing methods with both automatic evaluation metrics and human evaluation metrics.

Key facts. Cited by 64 papers in the map (data through 7 Oct 2026). First posted on arXiv 8 Sep 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 11; 2022: 12; 2023: 14; 2024: 12; 2025: 9; 2026: 6.

64 papers.

Sort:

By area:

  1. 2025Reasoning Models Don't Always Say What They Thinkcited by 388
  2. 2022Language Models are General-Purpose Interfacescited by 96
  3. 2023Do Models Explain Themselves? Counterfactual Simulatability of Natural Language Explanationscited by 85
  4. 2021e-ViL: A Dataset and Benchmark for Natural Language Explanations in Vision-Language Taskscited by 84
  5. 2021Few-Shot Self-Rationalization with Natural Language Promptscited by 83
  6. 2022NLX-GPT: A Model for Natural Language Explanations in Vision and Vision-Language Taskscited by 60
  7. 2024Explainable and Interpretable Multimodal Large Language Models: A Comprehensive Surveycited by 52
  8. 2021A Review on Explainability in Multimodal Deep Neural Netscited by 46
  9. 2021Diagnosing Vision-and-Language Navigation: What Really Matterscited by 36
  10. 2022MultiViz: Towards Visualizing and Understanding Multimodal Modelscited by 36
  11. 2021Local Interpretations for Explainable Natural Language Processing: A Surveycited by 31
  12. 2022Grounding Visual Representations with Texts for Domain Generalizationcited by 30
38 more
  1. 2023Beyond Task Performance: Evaluating and Reducing the Flaws of Large Multimodal Models with In-Context Learningcited by 30
  2. 2023Multimodal Explainable Artificial Intelligence: A Comprehensive Review of Methodological Advances and Future Research Directionscited by 25
  3. 2021Knowledge-Grounded Self-Rationalization via Extractive and Natural Language Explanationscited by 24
  4. 2024Do Vision & Language Decoders use Images and Text equally? How Self-consistent are their Explanations?cited by 23
  5. 2021Connecting Attributions and QA Model Behavior on Realistic Counterfactualscited by 20
  6. 2022CLEVR-X: A Visual Reasoning Dataset for Natural Language Explanationscited by 16
  7. 2024GazeXplain: Learning to Predict Natural Language Explanations of Visual Scanpathscited by 15
  8. 2021A First Look: Towards Explainable TextVQA Models via Visual and Textual Explanationscited by 14
  9. 2021Explainable Activity Recognition for Smart Home Systemscited by 14
  10. 2022REX: Reasoning-aware and Grounded Explanationcited by 12
  11. 2023Uni-NLX: Unifying Textual Explanations for Vision and Vision-Language Taskscited by 12
  12. 2025Explanation-Driven Counterfactual Testing for Faithfulness in Vision-Language Model Explanationscited by 12
  13. 2022Joint Answering and Explanation for Visual Commonsense Reasoningcited by 11
  14. 2022Explaining Deep Convolutional Neural Networks via Latent Visual-Semantic Filter Attentioncited by 11
  15. 2023Using Natural Language Explanations to Rescale Human Judgmentscited by 11
  16. 2024LLaVA Needs More Knowledge: Retrieval Augmented Natural Language Generation with Knowledge Graph for Explaining Thoracic Pathologiescited by 11
  17. 2022FRAME: Evaluating Rationale-Label Consistency Metrics for Free-Text Rationalescited by 10
  18. 2023S3C: Semi-Supervised VQA Natural Language Explanation via Self-Critical Learningcited by 10
  19. 2025ChartQA-X: Generating Explanations for Visual Chart Reasoningcited by 9
  20. 2022Chunk-aware Alignment and Lexical Constraint for Visual Entailment with Natural Language Explanationscited by 8
  21. 2024A Comprehensive Survey on Visual Question Answering Datasets and Algorithmscited by 8
  22. 2024Segmentation-guided Attention for Visual Question Answering from Remote Sensing Imagescited by 7
  23. 2024Faithful and Plausible Natural Language Explanations for Image Classification: A Pipeline Approachcited by 7
  24. 2022Harnessing the Power of Multi-Task Pretraining for Ground-Truth Level Natural Language Explanationscited by 6
  25. 2024The Illusion of Competence: Evaluating the Effect of Explanations on Users' Mental Models of Visual Question Answering Systemscited by 6
  26. 2025Explaining Similarity in Vision-Language Encoders with Weighted Banzhaf Interactionscited by 6
  27. 2022ILLUME: Rationalizing Vision-Language Models through Human Interactionscited by 5
  28. 2023Towards More Faithful Natural Language Explanation Using Multi-Level Contrastive Learning in VQAcited by 5
  29. 2023From Wrong To Right: A Recursive Approach Towards Vision-Language Explanationcited by 4
  30. 2023Towards a Unified Model for Generating Answers and Explanations in Visual Question Answeringcited by 3
  31. 2023Zero-shot Translation of Attention Patterns in VQA Models to Natural Languagecited by 3
  32. 2026Faithfulness Metrics Don't Measure Faithfulness: A Meta-Evaluation with Ground Truthcited by 3
  33. 2023Benchmarking Faithfulness: Towards Accurate Natural Language Explanations in Vision-Language Taskscited by 2
  34. 2023A Survey on Interpretable Cross-modal Reasoningcited by 2
  35. 2024Multimodal Rationales for Explainable Visual Question Answeringcited by 2
  36. 2025A Necessary Step toward Faithfulness: Measuring and Improving Consistency in Free-Text Explanationscited by 2
  37. 2025On the Risk of Misleading Reports: Diagnosing Textual Biases in Multimodal Clinical AIcited by 2
  38. 2021Designing Multimodal Datasets for NLP Challengescited by 1

50 most cited of 64 All 64 (JSON)