mlsanityData to 7 Oct 2026

Retrieval-Augmented Natural Language Reasoning for Explainable Visual Question Answering

arXiv 2408.17006 ↗First posted 30 Aug 2024cs.CVcited by 0

On the map: LLM reasoning → Multimodal reasoning

also in 1 more
Abstract (arXiv)

Visual Question Answering with Natural Language Explanation (VQA-NLE) task is challenging due to its high demand for reasoning-based inference. Recent VQA-NLE studies focus on enhancing model networks to amplify the model's reasoning capability but this approach is resource-consuming and unstable. In this work, we introduce a new VQA-NLE model, ReRe (Retrieval-augmented natural language Reasoning), using leverage retrieval information from the memory to aid in generating accurate answers and persuasive explanations without relying on complex networks and extra datasets. ReRe is an encoder-decoder architecture model using a pre-trained clip vision encoder and a pre-trained GPT-2 language model as a decoder. Cross-attention layers are added in the GPT-2 for processing retrieval features. ReRe outperforms previous methods in VQA accuracy and explanation score and shows improvement in NLE with more persuasive, reliability.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 25 of its 35 references that are in the map. First posted on arXiv 30 Aug 2024.

Builds on

25 of 35 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023From Wrong To Right: A Recursive Approach Towards Vision-Language Explanation21 Nov
  2. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering20 Sep

Other references · 23

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  3. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,690
  4. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  5. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  6. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  7. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,310
  8. 2021Improving language models by retrieving from trillions of tokenscited by 1,278
  9. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,013
  10. 2016SPICE: Semantic Propositional Image Caption Evaluationcited by 820
  11. 2018Multimodal Explanations: Justifying Decisions and Pointing to the Evidencecited by 179
  12. 2014A Multi-World Approach to Question Answering about Real-World Scenes based on Uncertain Inputcited by 112
11 more
  1. 2021e-ViL: A Dataset and Benchmark for Natural Language Explanations in Vision-Language Taskscited by 84
  2. 2022NLX-GPT: A Model for Natural Language Explanations in Vision and Vision-Language Taskscited by 60
  3. 2020Natural Language Rationales with Full-Stack Visual Reasoning: From Pixels to Semantic Frames to Commonsense Graphscited by 46
  4. 2022Retrieval-Augmented Transformer for Image Captioningcited by 38
  5. 2024Do You Remember? Dense Video Captioning with Cross-Modal Memory Retrievalcited by 36
  6. 2023Retrieving-to-Answer: Zero-Shot Video Question Answering with Frozen Large Language Modelscited by 27
  7. 2017Visual Question Answering with Memory-Augmented Networkscited by 25
  8. 2023S3C: Semi-Supervised VQA Natural Language Explanation via Self-Critical Learningcited by 10
  9. 2022Chunk-aware Alignment and Lexical Constraint for Visual Entailment with Natural Language Explanationscited by 8
  10. 2023Towards More Faithful Natural Language Explanation Using Multi-Level Contrastive Learning in VQAcited by 5
  11. 2023Towards a Unified Model for Generating Answers and Explanations in Visual Question Answeringcited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025SCRA-VQA: Summarized Caption-Rerank for Augmented Large Language Models in Visual Question Answering
  2. 2025ReAG: Reasoning-Augmented Generation for Knowledge-based Visual Question Answering
  3. 2024Natural Language Understanding and Inference with MLLM in Visual Question Answering: A Survey
  4. 2024Precision Empowers, Excess Distracts: Visual Question Answering With Dynamically Infused Knowledge In Language Models
  5. 2022A Unified End-to-End Retriever-Reader Framework for Knowledge-based VQA
  6. 2023Dynamic Clue Bottlenecks: Towards Interpretable-by-Design Visual Question Answering
  7. 2024Find The Gap: Knowledge Base Reasoning For Visual Question Answering
  8. 2026Look Twice: Training-Free Evidence Highlighting for Knowledge-based Visual Question Answering