mlsanityData to 14 Oct 2026

Augmented Vision-Language Models: A Systematic Review

arXiv 2507.22933 ↗First posted 24 Jul 2025cs.CL cs.AIcited by 1

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Recent advances in visual-language machine learning models have demonstrated exceptional ability to use natural language and understand visual scenes by training on large, unstructured datasets. However, this training paradigm cannot produce interpretable explanations for its outputs, requires retraining to integrate new information, is highly resource-intensive, and struggles with certain forms of logical reasoning. One promising solution involves integrating neural networks with external symbolic information systems, forming neural symbolic systems that can enhance reasoning and memory abilities. These neural symbolic systems provide more interpretable explanations to their outputs and the capacity to assimilate new information without extensive retraining. Utilizing powerful pre-trained Vision-Language Models (VLMs) as the core neural component, augmented by external systems, offers a pragmatic approach to realizing the benefits of neural-symbolic integration. This systematic literature review aims to categorize techniques through which visual-language understanding can be improved by interacting with external symbolic information systems.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 177 of its 290 references that are in the map. First posted on arXiv 24 Jul 2025.

Builds on

177 of 290 references.

Closest ancestors · 9

Same direction, newest first

  1. 2024Visual Sketchpad: Sketching as a Visual Chain of Thought for Multimodal Language Models13 Jun
  2. 2024LVLM-Interpret: An Interpretability Tool for Large Vision-Language Models3 Apr
  3. 2024SnapNTell: Enhancing Entity-Centric Visual Question Answering with Retrieval Augmented Multimodal LLM7 Mar
  4. 2023Visual Program Distillation: Distilling Tools and Programmatic Reasoning into Vision-Language Models5 Dec
  5. 2023Language Guided Visual Question Answering: Elevate Your Multimodal Language Model Using Knowledge-Enriched Prompts31 Oct
  6. 2023GraphAdapter: Tuning Vision-Language Models With Dual Knowledge Graph24 Sep
  7. 2023Can Vision-Language Models be a Good Guesser? Exploring VLMs for Times and Location Reasoning12 Jul
  8. 2023A Neural Divide-and-Conquer Reasoning Framework for Image Retrieval from Linguistically Complex Text3 May
  9. 2023Retrieval-based Knowledge Augmented Vision Language Pre-training27 Apr

Other references · 168

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  3. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  4. 2020Scaling Laws for Neural Language Modelscited by 6,586
  5. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,448
  6. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
  7. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 3,080
  8. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
  9. 2015VQA: Visual Question Answeringcited by 2,506
  10. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 2,006
  11. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,947
  12. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,729
38 more
  1. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,457
  2. 2018FEVER: a large-scale dataset for Fact Extraction and VERificationcited by 1,375
  3. 2023Gorilla: Large Language Model Connected with Massive APIscited by 1,332
  4. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,121
  5. 2016ConceptNet 5.5: An Open Multilingual Graph of General Knowledgecited by 1,067
  6. 2019How Can We Know What Language Models Know?cited by 959
  7. 2019Universal Adversarial Triggers for Attacking and Analyzing NLPcited by 722
  8. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 710
  9. 2023ViperGPT: Visual Inference via Python Execution for Reasoningcited by 603
  10. 2024GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Modelscited by 550
  11. 2022Visual Programming: Compositional visual reasoning without trainingcited by 516
  12. 2021Fast Model Editing at Scalecited by 498
  13. 2019CLEVRER: CoLlision Events for Video REpresentation and Reasoningcited by 484
  14. 2021Interpretable Machine Learning: Fundamental Principles and 10 Grand Challengescited by 367
  15. 2023Chameleon: Plug-and-Play Compositional Reasoning with Large Language Modelscited by 356
  16. 2021An Empirical Study of GPT-3 for Few-Shot Knowledge-Based VQAcited by 301
  17. 2024VideoAgent: A Memory-augmented Multimodal Agent for Video Understandingcited by 229
  18. 2023Can Pre-trained Vision and Language Models Answer Visual Information-Seeking Questions?cited by 207
  19. 2017Neural-Symbolic Learning and Reasoning: A Survey and Interpretationcited by 196
  20. 2020The Next Decade in AI: Four Steps Towards Robust Artificial Intelligencecited by 195
  21. 2023LLaVA-Plus: Learning to Use Tools for Creating Multimodal Agentscited by 193
  22. 2016FVQA: Fact-based Visual Question Answeringcited by 181
  23. 2023MMICL: Empowering Vision-language Model with Multi-Modal In-Context Learningcited by 172
  24. 2023Physically Grounded Vision-Language Models for Robotic Manipulationcited by 165
  25. 2023CRAFT: Customizing LLMs by Creating and Retrieving from Specialized Toolsetscited by 129
  26. 2023Knowledge-enhanced Visual-Language Pre-training on Chest Radiology Imagescited by 128
  27. 2021KAT: A Knowledge Augmented Transformer for Vision-and-Languagecited by 126
  28. 2020KRISP: Integrating Implicit and Symbolic Knowledge for Open-Domain Knowledge-Based VQAcited by 125
  29. 2022REVEAL: Retrieval-Augmented Visual-Language Pre-Training with Multi-Source Multimodal Knowledge Memorycited by 123
  30. 2019Learning by Abstraction: The Neural State Machinecited by 113
  31. 2024Is A Picture Worth A Thousand Words? Delving Into Spatial Reasoning for Vision Language Modelscited by 113
  32. 2019Human-Centered Tools for Coping with Imperfect Algorithms during Medical Decision-Makingcited by 109
  33. 2023Encyclopedic VQA: Visual questions about detailed properties of fine-grained categoriescited by 107
  34. 2019Scene Graph Generation with External Knowledge and Image Reconstructioncited by 102
  35. 2024ScreenAgent: A Vision Language Model-driven Computer Control Agentcited by 102
  36. 2022Retrieval Augmented Visual Question Answering with Outside Knowledgecited by 94
  37. 2022Super-CLEVR: A Virtual Benchmark to Diagnose Domain Robustness in Visual Reasoningcited by 92
  38. 2025VTool-R1: VLMs Learn to Think with Images via Reinforcement Learning on Multimodal Tool Usecited by 90

50 most cited of 168 All 168 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Pixelis: Reasoning in Pixels, from Seeing to Actingcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2022Deep Neural Networks for Visual Reasoning
  2. 2022Vision-Language Intelligence: Tasks, Representation Learning, and Large Models
  3. 2024Improving Fine-grained Visual Understanding in VLMs through Text-Only Training
  4. 2024Towards Interpreting Visual Information Processing in Vision-Language Models
  5. 2025Synthesizing Visual Concepts as Vision-Language Programs
  6. 2022VQA and Visual Reasoning: An Overview of Recent Datasets, Methods and Challenges
  7. 2023Vision-Language Models for Vision Tasks: A Survey
  8. 2024Exploring the Frontier of Vision-Language Models: A Survey of Current Methodologies and Future Directions