mlsanityData to 14 Oct 2026

Enhance Reasoning Ability of Visual-Language Models via Large Language Models

arXiv 2305.13267 ↗First posted 22 May 2023cs.CL cs.AIcited by 2

On the map: LLM reasoning → Multimodal reasoning

also in 1 more
Abstract (arXiv)

Pre-trained visual language models (VLM) have shown excellent performance in image caption tasks. However, it sometimes shows insufficient reasoning ability. In contrast, large language models (LLMs) emerge with powerful reasoning capabilities. Therefore, we propose a method called TReE, which transfers the reasoning ability of a large language model to a visual language model in zero-shot scenarios. TReE contains three stages: observation, thinking, and re-thinking. Observation stage indicates that VLM obtains the overall information of the relative image. Thinking stage combines the image information and task description as the prompt of the LLM, inference with the rationals. Re-Thinking stage learns from rationale and then inference the final result through VLM.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 34 of its 40 references that are in the map. First posted on arXiv 22 May 2023.

Builds on

34 of 40 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Action20 Mar
  2. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Models8 Mar
  3. 2023Multimodal Chain-of-Thought Reasoning in Language Models2 Feb

Other references · 31

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  4. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  5. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  6. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  7. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  8. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  9. 2021Learning to Prompt for Vision-Language Modelscited by 2,281
  10. 2022Conditional Prompt Learning for Vision-Language Modelscited by 1,433
  11. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,407
  12. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,121
19 more
  1. 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 715
  2. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 710
  3. 2022Automatic Chain of Thought Prompting in Large Language Modelscited by 688
  4. 2022MaPLe: Multi-modal Prompt Learningcited by 687
  5. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 647
  6. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  7. 2021DenseCLIP: Language-Guided Dense Prediction with Context-Aware Promptingcited by 532
  8. 2019Unified Vision-Language Pre-Training for Image Captioning and VQAcited by 494
  9. 2021MetaICL: Learning to Learn In Contextcited by 454
  10. 2021Unifying Vision-and-Language Tasks via Text Generationcited by 412
  11. 2023Larger language models do in-context learning differentlycited by 394
  12. 2022Teaching Small Language Models to Reasoncited by 284
  13. 2022VLP: A Survey on Vision-Language Pre-trainingcited by 159
  14. 2023Prompt, Generate, then Cache: Cascade of Foundation Models makes Strong Few-shot Learnerscited by 152
  15. 2022Rationale-Augmented Ensembles in Language Modelscited by 111
  16. 2020XGPT: Cross-modal Generative Pre-Training for Image Captioningcited by 51
  17. 2023Video ChatCaptioner: Towards Enriched Spatiotemporal Descriptionscited by 42
  18. 2023In-Context Learning with Many Demonstration Examplescited by 39
  19. 2022Multi-Stage Prompting for Knowledgeable Dialogue Generationcited by 22

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 0; 2026: 1.

2 papers.

  1. 2024Benchmarking VLMs' Reasoning About Persuasive Atypical Imagescited by 5
  2. 2026Enhancing Vision Language Models with Logic Reasoning for Situational Awarenesscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Enhancing Advanced Visual Reasoning Ability of Large Language Models
  2. 2024Zero-Shot Visual Reasoning by Vision-Language Models: Benchmarking and Analysis
  3. 2025Beyond Generation: Multi-Hop Reasoning for Factual Accuracy in Vision-Language Models
  4. 2025Decomposing Visual Classification: Assessing Tree-Based Reasoning in VLMs
  5. 2024Advancing Large Multi-modal Models with Explicit Chain-of-Reasoning and Visual Question Generation
  6. 2023ViCor: Bridging Visual Understanding and Commonsense Reasoning with Large Language Models
  7. 2025Visual Large Language Models for Generalized and Specialized Applications
  8. 2023VILA: On Pre-training for Visual Language Models