mlsanityData to 14 Oct 2026

From Wrong To Right: A Recursive Approach Towards Vision-Language Explanation

arXiv 2311.12391 ↗First posted 21 Nov 2023cs.CVcited by 4

On the map: LLM reasoning → Multimodal reasoning

also in 1 more
Abstract (arXiv)

Addressing the challenge of adapting pre-trained vision-language models for generating insightful explanations for visual reasoning tasks with limited annotations, we present ReVisE: a Recursive Visual Explanation algorithm. Our method iteratively computes visual features (conditioned on the text input), an answer, and an explanation, to improve the explanation quality step by step until the answer converges. We find that this multi-step approach guides the model to correct its own answers and outperforms single-step explanation generation. Furthermore, explanations generated by ReVisE also serve as valuable annotations for few-shot self-training. Our approach outperforms previous methods while utilizing merely 5% of the human-annotated explanations across 10 metrics, demonstrating up to a 4.2 and 1.3 increase in BLEU-1 score on the VCR and VQA-X datasets, underscoring the efficacy and data-efficiency of our method.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 44 of its 65 references that are in the map. First posted on arXiv 21 Nov 2023.

Builds on

44 of 65 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering20 Sep

Other references · 43

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  3. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,733
  4. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  5. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  6. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  7. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  8. 2015VQA: Visual Question Answeringcited by 2,506
  9. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,375
  10. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,046
  11. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,619
  12. 2020Big Self-Supervised Models are Strong Semi-Supervised Learnerscited by 1,380
31 more
  1. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,362
  2. 2019Self-training with Noisy Student improves ImageNet classificationcited by 1,311
  3. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,230
  4. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,027
  5. 2016SPICE: Semantic Propositional Image Caption Evaluationcited by 824
  6. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 766
  7. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 727
  8. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 710
  9. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 647
  10. 2018From Recognition to Cognition: Visual Commonsense Reasoningcited by 556
  11. 2019Using Self-Supervised Learning Can Improve Model Robustness and Uncertaintycited by 514
  12. 2021VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Expertscited by 425
  13. 2019Confidence Regularized Self-Trainingcited by 419
  14. 2021MERLOT: Multimodal Neural Script Knowledge Modelscited by 311
  15. 2023Faithful Chain-of-Thought Reasoningcited by 288
  16. 2020UNIMO: Towards Unified-Modal Understanding and Generation via Cross-Modal Contrastive Learningcited by 239
  17. 2018Multimodal Explanations: Justifying Decisions and Pointing to the Evidencecited by 179
  18. 2019Vision-Language Navigation with Self-Supervised Auxiliary Reasoning Taskscited by 179
  19. 2016Generating Visual Explanationscited by 158
  20. 2023Debiasing Vision-Language Models via Biased Promptscited by 149
  21. 2020Improving One-stage Visual Grounding by Recursive Sub-query Constructioncited by 138
  22. 2021e-ViL: A Dataset and Benchmark for Natural Language Explanations in Vision-Language Taskscited by 84
  23. 2019Learning to Self-Train for Semi-Supervised Few-Shot Classificationcited by 83
  24. 2018Faithful Multimodal Explanation for Visual Question Answeringcited by 64
  25. 2019Self-Critical Reasoning for Robust Visual Question Answeringcited by 61
  26. 2022NLX-GPT: A Model for Natural Language Explanations in Vision and Vision-Language Taskscited by 60
  27. 2020e-SNLI-VE: Corrected Visual-Textual Entailment with Natural Language Explanationscited by 31
  28. 2017Learning by Asking Questionscited by 25
  29. 2021Revisiting Self-Training for Few-Shot Learning of Language Modelcited by 23
  30. 2020Image Captioning as an Assistive Technology: Lessons Learned from VizWiz 2020 Challengecited by 17
  31. 2022Harnessing the Power of Multi-Task Pretraining for Ground-Truth Level Natural Language Explanationscited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 1; 2026: 1.

4 papers.

  1. 2024When Can LLMs Actually Correct Their Own Mistakes? A Critical Survey of Self-Correction of LLMscited by 240
  2. 2025Analyzing Finetuning Representation Shift for Multimodal LLMs Steeringcited by 16
  3. 2026Architectural Sampling: Test-Time Scaling via Computational Diversity in Frozen Vision-Language Modelscited by 1
  4. 2024Retrieval-Augmented Natural Language Reasoning for Explainable Visual Question Answeringcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Evaluating Explanation-Driven Vision-Language Reasoning via Generation Order Interventions
  2. 2026Improving Reasoning in Vision-Language Models via Perception Verified Self-Training
  3. 2026Improving Visual Reasoning with Iterative Evidence Refinement
  4. 2025See, Think, Learn: A Self-Taught Multimodal Reasoner
  5. 2025Unbiased Visual Reasoning with Controlled Visual Inputs
  6. 2026Logit Lens Supervision for Patch-Level Explanations in Vision-Language Models
  7. 2023Uni-NLX: Unifying Textual Explanations for Vision and Vision-Language Tasks
  8. 2024From the Least to the Most: Building a Plug-and-Play Visual Reasoner via Data Synthesis