A Survey on Interpretable Cross-modal Reasoning
Not on the map. Why?
Abstract (arXiv)
In recent years, cross-modal reasoning (CMR), the process of understanding and reasoning across different modalities, has emerged as a pivotal area with applications spanning from multimedia analysis to healthcare diagnostics. As the deployment of AI systems becomes more ubiquitous, the demand for transparency and comprehensibility in these systems' decision-making processes has intensified. This survey delves into the realm of interpretable cross-modal reasoning (I-CMR), where the objective is not only to achieve high predictive performance but also to provide human-understandable explanations for the results. This survey presents a comprehensive overview of the typical methods with a three-level taxonomy for I-CMR. Furthermore, this survey reviews the existing CMR datasets with annotations for explanations. Finally, this survey summarizes the challenges for I-CMR and discusses potential future directions. In conclusion, this survey aims to catalyze the progress of this emerging research area by providing researchers with a panoramic and comprehensive perspective, illuminating the state of the art and discerning the opportunities. The summarized methods, datasets, and other resources are available at https://github.com/ZuyiZhou/Awesome-Interpretable-Cross-modal-Reasoning.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 76 of its 117 references that are in the map. First posted on arXiv 5 Sep 2023.
Builds on
76 of 117 references.
References · 76
- 2020Language Models are Few-Shot Learnerscited by 32,199
- 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
- 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
- 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,690
- 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
- 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
- 2015VQA: Visual Question Answeringcited by 2,489
- 2017FiLM: Visual Reasoning with a General Conditioning Layercited by 2,286
- 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
- 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
- 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
- 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,026
38 more
- 2017Multimodal Machine Learning: A Survey and Taxonomycited by 1,004
- 2023Multimodal Chain-of-Thought Reasoning in Language Modelscited by 740
- 2018From Recognition to Cognition: Visual Commonsense Reasoningcited by 553
- 2018Explaining Explanations: An Overview of Interpretability of Machine Learningcited by 508
- 2022Visual Programming: Compositional visual reasoning without trainingcited by 507
- 2022Multimodal Learning with Transformers: A Surveycited by 399
- 2015Neural Module Networkscited by 375
- 2018Neural-Symbolic VQA: Disentangling Reasoning from Vision and Language Understandingcited by 304
- 2018Touchdown: Natural Language Navigation and Spatial Reasoning in Visual Street Environmentscited by 238
- 2019Full-Gradient Representation for Neural Network Visualizationcited by 189
- 2019Visual Semantic Reasoning for Image-Text Matchingcited by 185
- 2016FVQA: Fact-based Visual Question Answeringcited by 181
- 2018Multimodal Explanations: Justifying Decisions and Pointing to the Evidencecited by 179
- 2020Improving Vision-and-Language Navigation with Image-Text Pairs from the Webcited by 177
- 2022X-Pool: Cross-Modal Language-Video Attention for Text-Video Retrievalcited by 127
- 2017Learning to Reason: End-to-End Module Networks for Visual Question Answeringcited by 125
- 2019Relation-Aware Graph Attention Network for Visual Question Answeringcited by 119
- 2022From Images to Textual Prompts: Zero-shot VQA with Frozen Large Language Modelscited by 114
- 2022End-to-End Multimodal Fact-Checking and Explanation Generation: A Challenging Dataset and Modelscited by 105
- 2018Explainable and Explicit Visual Reasoning over Scene Graphscited by 103
- 2019MUREL: Multimodal Relational Reasoning for Visual Question Answeringcited by 68
- 2018Faithful Multimodal Explanation for Visual Question Answeringcited by 64
- 2015Hinge-Loss Markov Random Fields and Probabilistic Soft Logiccited by 63
- 2023T-SciQ: Teaching Multimodal Chain-of-Thought Reasoning via Mixed Large Language Model Signals for Science Question Answeringcited by 59
- 2023NS3D: Neuro-Symbolic Grounding of 3D Objects and Relationscited by 57
- 2018Explainable Neural Computation via Stack Neural Module Networkscited by 54
- 2020Mucko: Multi-Layer Cross-Modal Knowledge Reasoning for Fact-based Visual Question Answeringcited by 51
- 2018VQA-E: Explaining, Elaborating, and Enhancing Your Answers for Visual Questionscited by 50
- 2019Learning Cross-modal Context Graph for Visual Groundingcited by 49
- 2020VQA-LOL: Visual Question Answering under the Lens of Logiccited by 48
- 2022Grounding Answers for Visual Questions Asked by Visually Impaired Peoplecited by 47
- 2022A survey of multimodal deep generative modelscited by 47
- 2020Natural Language Rationales with Full-Stack Visual Reasoning: From Pixels to Semantic Frames to Commonsense Graphscited by 46
- 2018Learning Conditioned Graph Structures for Interpretable Visual Question Answeringcited by 45
- 2023Video-Text as Game Players: Hierarchical Banzhaf Interaction for Cross-Modal Representation Learningcited by 45
- 2019Meta Module Network for Compositional Visual Reasoningcited by 39
- 2023Decoding the Underlying Meaning of Multimodal Hateful Memescited by 38
- 2022MuKEA: Multimodal Knowledge Extraction and Accumulation for Knowledge-based Visual Question Answeringcited by 36
50 most cited of 76 All 76 (JSON)
Built on by
Citing papers per year: 2024: 2.
2 papers.