mlsanityData to 14 Oct 2026

Prism: A Framework for Decoupling and Assessing the Capabilities of VLMs

arXiv 2406.14544 ↗First posted 20 Jun 2024cs.CV cs.CLcited by 32

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Vision Language Models (VLMs) demonstrate remarkable proficiency in addressing a wide array of visual questions, which requires strong perception and reasoning faculties. Assessing these two competencies independently is crucial for model refinement, despite the inherent difficulty due to the intertwined nature of seeing and reasoning in existing VLMs. To tackle this issue, we present Prism, an innovative framework designed to disentangle the perception and reasoning processes involved in visual question solving. Prism comprises two distinct stages: a perception stage that utilizes a VLM to extract and articulate visual information in textual form, and a reasoning stage that formulates responses based on the extracted visual information using a Large Language Model (LLM). This modular design enables the systematic comparison and assessment of both proprietary and open-source VLM for their perception and reasoning strengths. Our analytical framework provides several valuable insights, underscoring Prism's potential as a cost-effective solution for vision-language tasks. By combining a streamlined VLM focused on perception with a powerful LLM tailored for reasoning, Prism achieves superior results in general vision-language tasks while substantially cutting down on training and operational expenses. Quantitative evaluations show that Prism, when configured with a vanilla 2B LLaVA and freely accessible GPT-3.5, delivers performance on par with VLMs 10 × larger on the rigorous multimodal benchmark MMStar. The project is released at: https://github.com/SparksJoe/Prism.

Key facts. Cited by 32 papers in the map (data through 14 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 50 of its 66 references that are in the map. First posted on arXiv 20 Jun 2024.

Builds on

50 of 66 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?29 Mar
  2. 2024How Far Are We from Intelligent Visual Deductive Reasoning?7 Mar
  3. 2024ALLaVA: Harnessing GPT4V-Synthesized Data for Lite Vision-Language Models18 Feb
  4. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts3 Oct
  5. 2023Kosmos-2: Grounding Multimodal Large Language Models to the World26 Jun
  6. 2022Flamingo: a Visual Language Model for Few-Shot Learning29 Apr

Other references · 44

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  6. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  7. 2023Visual Instruction Tuningcited by 7,869
  8. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  9. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  10. 2023Qwen Technical Reportcited by 3,344
  11. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  12. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
32 more
  1. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
  2. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  3. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,141
  4. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
  5. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,729
  6. 2019Towards VQA Models That Can Readcited by 1,718
  7. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,652
  8. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,555
  9. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,479
  10. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,317
  11. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,216
  12. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,121
  13. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,050
  14. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,040
  15. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,031
  16. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,005
  17. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 940
  18. 2016A Diagram Is Worth A Dozen Imagescited by 804
  19. 2023EVA-CLIP: Improved Training Techniques for CLIP at Scalecited by 798
  20. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMscited by 735
  21. 2024DeepSeek-VL: Towards Real-World Vision-Language Understandingcited by 733
  22. 2024BLINK: Multimodal Large Language Models Can See but Not Perceivecited by 550
  23. 2022Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionalitycited by 529
  24. 2023Generative Multimodal Models are In-Context Learnerscited by 446
  25. 2018nocaps: novel object captioning at scalecited by 444
  26. 2024InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Modelcited by 326
  27. 2022Visual Spatial Reasoningcited by 293
  28. 2023LVLM-eHub: A Comprehensive Evaluation Benchmark for Large Vision-Language Modelscited by 217
  29. 2024Reka Core, Flash, and Edge: A Series of Powerful Multimodal Language Modelscited by 67
  30. 2024CounterCurate: Enhancing Physical and Semantic Visio-Linguistic Compositional Reasoning via Counterfactual Examplescited by 35
  31. 2023InfiMM-Eval: Complex Open-Ended Reasoning Evaluation For Multi-Modal Large Language Modelscited by 14
  32. 2024Diagnosing the Compositional Knowledge of Vision Language Models from a Game-Theoretic Viewcited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 9; 2025: 12; 2026: 11.

32 papers.

Sort:

By area:

  1. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
  2. 2024LLaVA-CoT: Let Vision Language Models Reason Step-by-Stepcited by 454
  3. 2024PyramidDrop: Accelerating Your Large Vision-Language Models via Pyramid Visual Redundancy Reductioncited by 294
  4. 2024MME-Survey: A Comprehensive Survey on Evaluation of Multimodal LLMscited by 72
  5. 2025Steer LLM Latents for Hallucination Detectioncited by 50
  6. 2025CapRL: Stimulating Dense Image Caption Capabilities via Reinforcement Learningcited by 38
  7. 2025VisualWebInstruct: Scaling up Multimodal Instruction Data through Web Searchcited by 35
  8. 2025ReviewAgents: Bridging the Gap Between Human and AI-Generated Paper Reviewscited by 32
  9. 2025MindOmni: Unleashing Reasoning Generation in Vision Language Models with RGPOcited by 30
  10. 2024Deciphering Cross-Modal Alignment in Large Vision-Language Models with Modality Integration Ratecited by 20
  11. 2024POINTS: Improving Your Vision-language Model with Affordable Strategiescited by 17
  12. 2024Bongard in Wonderland: Visual Puzzles that Still Make AI Go Mad?cited by 16
20 more
  1. 2025ScaleCap: Inference-Time Scalable Image Captioning via Dual-Modality Debiasingcited by 15
  2. 2025Creation-MMBench: Assessing Context-Aware Creative Intelligence in MLLMcited by 14
  3. 2024Towards Building Specialized Generalist AI with System 1 and System 2 Fusioncited by 8
  4. 2026CCCaption: Dual-Reward Reinforcement Learning for Complete and Correct Image Captioningcited by 8
  5. 2026CapRL++: Unified Reinforcement Learning with Verifiable Rewards for Dense Image and Video Captioningcited by 8
  6. 2024Beyond Captioning: Task-Specific Prompting for Improved VLM Performance in Mathematical Reasoningcited by 5
  7. 2025EVADE-Bench: Multimodal Benchmark for Evaluating and Enhancing Evasive Content Detectioncited by 5
  8. 2025VisualSimpleQA: A Benchmark for Decoupled Evaluation of Large Vision-Language Models in Fact-Seeking Question Answeringcited by 4
  9. 2025Caption This, Reason That: VLMs Caught in the Middlecited by 4
  10. 2025Benchmarking Vision Language Models on German Factual Datacited by 2
  11. 2026BalCapRL: A Balanced Framework for RL-Based MLLM Image Captioningcited by 2
  12. 2026Momentum-Coupled Rubric Adaptation for Detailed Image Captioningcited by 2
  13. 2025Exploring Primitive Visual Measurement Understanding and the Role of Output Format in Learning in Vision-Language Modelscited by 1
  14. 2026VCap: Hypergeometric Rewards for Weak-to-Strong Visual Captioningcited by 1
  15. 2026Tool-IQA: Augmenting Image Quality Assessment with Simple Toolscited by 1
  16. 2026Can VLMs Reason Robustly? A Neuro-Symbolic Investigationcited by 0
  17. 2026Back to the Barn with LLAMAs: Evolving Pretrained LLM Backbones in Finetuning Vision Language Modelscited by 0
  18. 2026CRAG-MM-Diagnostics: Enabling Stage-Wise Analysis of Knowledge-Intensive VQAcited by 0
  19. 2026A Reconstruction-Based Framework for Caption Evaluation Beyond Reference Captionscited by 0
  20. 2026Seeing and Solving Are Not Enough for Vision-Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026PRISM: Perception Reasoning Interleaved for Sequential Decision Making
  2. 2024Prismatic VLMs: Investigating the Design Space of Visually-Conditioned Language Models
  3. 2023Large Language Models are Visual Reasoning Coordinators
  4. 2026Evaluating VQA in Vision Language Models using Cooperative Principles
  5. 2026From Perception to Integration: Revisiting the Internal Dynamics of Reasoning in Vision-Language Models
  6. 2025VisRes Bench: On Evaluating the Visual Reasoning Capabilities of VLMs
  7. 2026From Seeing to Thinking: Decoupling Perception and Reasoning Improves Post-Training of Vision-Language Models
  8. 2024Zero-Shot Visual Reasoning by Vision-Language Models: Benchmarking and Analysis