A Survey of State of the Art Large Vision Language Models: Alignment, Benchmark, Evaluations and Challenges
On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning
also in 2 more
Abstract (arXiv)
Large vision-language models (VLMs) have evolved rapidly from contrastive image-text encoders and adapter-based assistants into natively multimodal foundation models that support long-context reasoning, agentic workflows, and, increasingly, unified perception-generation-action loops. Since 2025, the frontier has consolidated around a small number of large pretraining families, including GPT, Gemini, Claude, Grok, Qwen, Gemma, DeepSeek, Kimi, and MiniMax, while evaluation has shifted from short-form visual question answering toward spatial, temporal, embodied, and calibration-sensitive benchmarks. In this survey, we provide an updated overview of VLMs through 2026 with an emphasis on three developments. First, we trace the architectural evolution from contrastive or bridged two-tower models to LLM-backbone adapter models, native multimodal-input models, omni-modal unified input/output models, and emerging world-action models. Second, we summarize how benchmarks expanded from classical OCR, VQA, and chart understanding toward long-video reasoning, embodied evaluation, reward-model judging, and domain-specific decision making. Third, we review the growing role of post-training, including supervised fine-tuning, preference optimization, and reinforcement learning methods such as GRPO-style multimodal alignment. Rather than exhaustively listing every release, we focus on representative frontier families, influential benchmarks, and the major open challenges that remain in hallucination, safety, efficiency, data quality, and multimodal alignment.
Key facts. Cited by 46 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 145 of its 226 references that are in the map. First posted on arXiv 4 Jan 2025.
Builds on
145 of 226 references.
Closest ancestors · 9
Same direction, newest first
- 2024NaturalBench: Evaluating Vision-Language Models on Natural Adversarial Samples18 Oct
- 2024Benchmarking Vision Language Models for Cultural Understanding15 Jul
- 2024An Introduction to Vision-Language Modeling27 May
- 2024Enhancing Visual-Language Modality Alignment in Large Vision Language Models via Self-Improvement24 May
- 2024Are We on the Right Way for Evaluating Large Vision-Language Models?29 Mar
- 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond24 Aug
- 2022Flamingo: a Visual Language Model for Few-Shot Learning29 Apr
- 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation28 Jan
- 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervision24 Aug
Other references · 136
- 2015Deep Residual Learning for Image Recognitioncited by 51,809
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
- 2020Language Models are Few-Shot Learnerscited by 32,199
- 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
- 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
- 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
- 2020Measuring Massive Multitask Language Understandingcited by 7,768
- 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
- 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
- 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
- 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
- 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
38 more
- 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
- 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
- 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
- 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,388
- 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
- 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
- 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
- 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 1,908
- 2023VBench: Comprehensive Benchmark Suite for Video Generative Modelscited by 1,755
- 2019Towards VQA Models That Can Readcited by 1,696
- 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
- 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
- 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,449
- 2018World Modelscited by 1,444
- 2016Context Encoders: Feature Learning by Inpaintingcited by 1,424
- 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,375
- 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,288
- 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
- 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
- 2020Leveraging Passage Retrieval with Generative Models for Open Domain Question Answeringcited by 1,203
- 2019Habitat: A Platform for Embodied AI Researchcited by 1,163
- 2023GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignmentcited by 946
- 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 929
- 2024WorldSimBench: Towards Video Generation Models as World Simulatorscited by 867
- 2017AI2-THOR: An Interactive 3D Environment for Visual AIcited by 850
- 2019Self-Supervised Learning of Pretext-Invariant Representationscited by 806
- 2016A Diagram Is Worth A Dozen Imagescited by 793
- 2017Reading Wikipedia to Answer Open-Domain Questionscited by 777
- 2023AGIEval: A Human-Centric Benchmark for Evaluating Foundation Modelscited by 701
- 2023C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Modelscited by 694
- 2023CogAgent: A Visual Language Model for GUI Agentscited by 692
- 2023VILA: On Pre-training for Visual Language Modelscited by 686
- 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)cited by 677
- 2023EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understandingcited by 666
- 2018Object Hallucination in Image Captioningcited by 661
- 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 657
- 2023Bias and Fairness in Large Language Models: A Surveycited by 626
- 2023GAIA-1: A Generative World Model for Autonomous Drivingcited by 618
50 most cited of 136 All 136 (JSON)
Built on by
Citing papers per year: 2025: 29; 2026: 17.
46 papers.
Sort:
By area:
- 2025Thinking with Images for Multimodal Reasoning: Foundations, Methods, and Future Frontierscited by 193
- 2026Graph-of-Skills: Dependency-Aware Structural Retrieval for Massive Agent Skillscited by 53
- 2025ColorBench: Can VLMs See and Understand the Colorful World? A Comprehensive Benchmark for Color Perception, Reasoning, and Robustnesscited by 28
- 2025DreamPRM: Domain-Reweighted Process Reward Model for Multimodal Reasoningcited by 15
- 2025SpinBench: Perspective and Rotation as a Lens on Spatial Reasoning in VLMscited by 12
- 2025VACoT: Rethinking Visual Data Augmentation with VLMscited by 8
- 2025CausalVLBench: Benchmarking Visual Causal Reasoning in Large Vision-Language Modelscited by 7
- 2025ORIGAMISPACE: Benchmarking Multimodal LLMs in Multi-Step Spatial Reasoning with Mathematical Constraintscited by 7
- 2025Small Drafts, Big Verdict: Information-Intensive Visual Reasoning via Speculationcited by 6
- 2026Collaborative Multi-Agent Scripts Generation for Enhancing Imperfect-Information Reasoning in Murder Mystery Gamescited by 6
- 2025SemVink: Advancing VLMs' Semantic Understanding of Optical Illusions via Visual Global Thinkingcited by 5
- 2025MASS: Motion-Aware Spatial-Temporal Grounding for Physics Reasoning and Comprehension in Vision-Language Modelscited by 5
34 more
- 2025Limits and Gains of Test-Time Scaling in Vision-Language Reasoningcited by 5
- 2025HUMORCHAIN: Theory-Guided Multi-Stage Reasoning for Interpretable Multimodal Humor Generationcited by 4
- 2026MARE: Multimodal Alignment and Reinforcement for Explainable Deepfake Detection via Vision-Language Modelscited by 4
- 2025VisChainBench: A Benchmark for Multi-Turn, Multi-Image Visual Reasoning Beyond Language Priorscited by 3
- 2025Modeling Beyond MOS: Quality Assessment Models Must Integrate Context, Reasoning, and Multimodalitycited by 2
- 2025FC-MIR: A Mobile Screen Awareness Framework for Intent-Aware Recommendation based on Frame-Compressed Multimodal Trajectory Reasoningcited by 2
- 2026Thinking in Structures: Evaluating Spatial Intelligence in Constraint-Governed Spacescited by 2
- 2026ArchSIBench: Benchmarking the Architectural Spatial Intelligence of Vision-Language Modelscited by 2
- 2025BYO-Eval: Build Your Own Dataset for Fine-Grained Visual Assessment of Multimodal Language Modelscited by 1
- 2025COREVQA: A Crowd Observation and Reasoning Entailment Visual Question Answering Benchmarkcited by 1
- 2025FOUNDv2: Learning Unified User Quantized Tokenizers for User Representationcited by 1
- 2025Understanding Space Is Rocket Science -- Only Top Reasoning Models Can Solve Spatial Understanding Taskscited by 1
- 2025AgentCaster: Reasoning-Guided Tornado Forecastingcited by 1
- 2025COVLM-RL: Critical Object-Oriented Reasoning for Autonomous Driving Using VLM-Guided Reinforcement Learningcited by 1
- 2025From Building Blocks to Planning: Multi-Step Spatial Reasoning in LLMs with Reinforcement Learningcited by 1
- 2026It's Time to Get It Right: Improving Analog Clock Reading and Clock-Hand Spatial Reasoning in Vision-Language Modelscited by 1
- 2026Almieyar-Oryx-BloomBench: A Bilingual Multimodal Benchmark for Cognitively Informed Evaluation of Vision-Language Modelscited by 1
- 2026MIRROR: Learning from the Other View for Multi-Modal Reasoningcited by 1
- 2025Elevating Visual Question Answering through Implicitly Learned Reasoning Pathways in LVLMscited by 0
- 2025CARMA: Context-Aware Situational Grounding of Human-Robot Group Interactions by Combining Vision-Language Models with Object and Action Recognitioncited by 0
- 2025Ascending the Infinite Ladder: Benchmarking Spatial Deformation Reasoning in Vision-Language Modelscited by 0
- 2025Decomposing Visual Classification: Assessing Tree-Based Reasoning in VLMscited by 0
- 2025Rule-Based Reinforcement Learning for Document Image Classification with Vision Language Modelscited by 0
- 2025AgroCoT: A Chain-of-Thought Benchmark for Evaluating Reasoning in Vision-Language Models for Agriculturecited by 0
- 2025Rethinking Fine-Tuning: Unlocking Hidden Capabilities in Vision-Language Modelscited by 0
- 2026Lightweight Visual Reasoning for Socially-Aware Robotscited by 0
- 2026Spatial Colour Mixing Illusions as a Perception Stress Test for Vision-Language Modelscited by 0
- 2026MERGE: Guided Vision-Language Models for Multi-Actor Event Reasoning and Grounding in Human-Robot Interactioncited by 0
- 2026Revealing Multi-View Hallucination in Large Vision-Language Modelscited by 0
- 2026HyperGVL: Benchmarking and Improving Large Vision-Language Models in Hypergraph Understanding and Reasoningcited by 0
- 2026On the Cultural Anachronism and Temporal Reasoning in Vision Language Modelscited by 0
- 2026IVR-R1: Refining Trajectories through Iterative Visual-Grounded Reasoning in Reinforcement Learningcited by 0
- 2026ExpArt-KG: Artwork Image Description Generation through Iterative Exploration of Knowledge Graphscited by 0
- 2026Benchmarking Vision-Language Models for Automated Pathology Diagnosis and Report Generationcited by 0
- 2026Benchmarking Vision-Language Models for Automated Pathology Diagnosis and Report Generation1 Sep 2026
- 2026ExpArt-KG: Artwork Image Description Generation through Iterative Exploration of Knowledge Graphs1 Sep 2026
- 2026MIRROR: Learning from the Other View for Multi-Modal Reasoning23 Jul 2026
- 2026Almieyar-Oryx-BloomBench: A Bilingual Multimodal Benchmark for Cognitively Informed Evaluation of Vision-Language Models4 Jun 2026
- 2026ArchSIBench: Benchmarking the Architectural Spatial Intelligence of Vision-Language Models20 May 2026
- 2026IVR-R1: Refining Trajectories through Iterative Visual-Grounded Reasoning in Reinforcement Learning18 May 2026
- 2026On the Cultural Anachronism and Temporal Reasoning in Vision Language Models14 May 2026
- 2026HyperGVL: Benchmarking and Improving Large Vision-Language Models in Hypergraph Understanding and Reasoning17 Apr 2026
- 2026Collaborative Multi-Agent Scripts Generation for Enhancing Imperfect-Information Reasoning in Murder Mystery Games13 Apr 2026
- 2026Graph-of-Skills: Dependency-Aware Structural Retrieval for Massive Agent Skills7 Apr 2026
- 2026Revealing Multi-View Hallucination in Large Vision-Language Models25 Mar 2026
- 2026MERGE: Guided Vision-Language Models for Multi-Actor Event Reasoning and Grounding in Human-Robot Interaction19 Mar 2026
34 more
- 2026It's Time to Get It Right: Improving Analog Clock Reading and Clock-Hand Spatial Reasoning in Vision-Language Models9 Mar 2026
- 2026Spatial Colour Mixing Illusions as a Perception Stress Test for Vision-Language Models6 Mar 2026
- 2026Lightweight Visual Reasoning for Socially-Aware Robots4 Mar 2026
- 2026Thinking in Structures: Evaluating Spatial Intelligence in Constraint-Governed Spaces8 Feb 2026
- 2026MARE: Multimodal Alignment and Reinforcement for Explainable Deepfake Detection via Vision-Language Models28 Jan 2026
- 2025From Building Blocks to Planning: Multi-Step Spatial Reasoning in LLMs with Reinforcement Learning31 Dec 2025
- 2025Rethinking Fine-Tuning: Unlocking Hidden Capabilities in Vision-Language Models28 Dec 2025
- 2025FC-MIR: A Mobile Screen Awareness Framework for Intent-Aware Recommendation based on Frame-Compressed Multimodal Trajectory Reasoning22 Dec 2025
- 2025Limits and Gains of Test-Time Scaling in Vision-Language Reasoning11 Dec 2025
- 2025COVLM-RL: Critical Object-Oriented Reasoning for Autonomous Driving Using VLM-Guided Reinforcement Learning10 Dec 2025
- 2025VisChainBench: A Benchmark for Multi-Turn, Multi-Image Visual Reasoning Beyond Language Priors7 Dec 2025
- 2025VACoT: Rethinking Visual Data Augmentation with VLMs2 Dec 2025
- 2025AgroCoT: A Chain-of-Thought Benchmark for Evaluating Reasoning in Vision-Language Models for Agriculture28 Nov 2025
- 2025ORIGAMISPACE: Benchmarking Multimodal LLMs in Multi-Step Spatial Reasoning with Mathematical Constraints23 Nov 2025
- 2025MASS: Motion-Aware Spatial-Temporal Grounding for Physics Reasoning and Comprehension in Vision-Language Models23 Nov 2025
- 2025HUMORCHAIN: Theory-Guided Multi-Stage Reasoning for Interpretable Multimodal Humor Generation21 Nov 2025
- 2025Small Drafts, Big Verdict: Information-Intensive Visual Reasoning via Speculation23 Oct 2025
- 2025AgentCaster: Reasoning-Guided Tornado Forecasting2 Oct 2025
- 2025SpinBench: Perspective and Rotation as a Lens on Spatial Reasoning in VLMs29 Sep 2025
- 2025Rule-Based Reinforcement Learning for Document Image Classification with Vision Language Models26 Sep 2025
- 2025Decomposing Visual Classification: Assessing Tree-Based Reasoning in VLMs10 Sep 2025
- 2025Understanding Space Is Rocket Science -- Only Top Reasoning Models Can Solve Spatial Understanding Tasks2 Sep 2025
- 2025FOUNDv2: Learning Unified User Quantized Tokenizers for User Representation1 Aug 2025
- 2025COREVQA: A Crowd Observation and Reasoning Entailment Visual Question Answering Benchmark17 Jul 2025
- 2025Ascending the Infinite Ladder: Benchmarking Spatial Deformation Reasoning in Vision-Language Models1 Jul 2025
- 2025Thinking with Images for Multimodal Reasoning: Foundations, Methods, and Future Frontiers30 Jun 2025
- 2025CARMA: Context-Aware Situational Grounding of Human-Robot Group Interactions by Combining Vision-Language Models with Object and Action Recognition25 Jun 2025
- 2025BYO-Eval: Build Your Own Dataset for Fine-Grained Visual Assessment of Multimodal Language Models5 Jun 2025
- 2025SemVink: Advancing VLMs' Semantic Understanding of Optical Illusions via Visual Global Thinking3 Jun 2025
- 2025DreamPRM: Domain-Reweighted Process Reward Model for Multimodal Reasoning26 May 2025
- 2025Modeling Beyond MOS: Quality Assessment Models Must Integrate Context, Reasoning, and Multimodality26 May 2025
- 2025CausalVLBench: Benchmarking Visual Causal Reasoning in Large Vision-Language Models21 May 2025
- 2025ColorBench: Can VLMs See and Understand the Colorful World? A Comprehensive Benchmark for Color Perception, Reasoning, and Robustness10 Apr 2025
- 2025Elevating Visual Question Answering through Implicitly Learned Reasoning Pathways in LVLMs18 Mar 2025
- 2025Thinking with Images for Multimodal Reasoning: Foundations, Methods, and Future Frontierscited by 193
- 2025ColorBench: Can VLMs See and Understand the Colorful World? A Comprehensive Benchmark for Color Perception, Reasoning, and Robustnesscited by 28
- 2025DreamPRM: Domain-Reweighted Process Reward Model for Multimodal Reasoningcited by 15
- 2025SpinBench: Perspective and Rotation as a Lens on Spatial Reasoning in VLMscited by 12
- 2025VACoT: Rethinking Visual Data Augmentation with VLMscited by 8
- 2025CausalVLBench: Benchmarking Visual Causal Reasoning in Large Vision-Language Modelscited by 7
- 2025ORIGAMISPACE: Benchmarking Multimodal LLMs in Multi-Step Spatial Reasoning with Mathematical Constraintscited by 7
- 2025Small Drafts, Big Verdict: Information-Intensive Visual Reasoning via Speculationcited by 6
- 2026Collaborative Multi-Agent Scripts Generation for Enhancing Imperfect-Information Reasoning in Murder Mystery Gamescited by 6
- 2025SemVink: Advancing VLMs' Semantic Understanding of Optical Illusions via Visual Global Thinkingcited by 5
- 2025MASS: Motion-Aware Spatial-Temporal Grounding for Physics Reasoning and Comprehension in Vision-Language Modelscited by 5
- 2025Limits and Gains of Test-Time Scaling in Vision-Language Reasoningcited by 5
12 most cited of 36 All 36 (JSON)
- 2026MIRROR: Learning from the Other View for Multi-Modal Reasoning23 Jul 2026
- 2026Almieyar-Oryx-BloomBench: A Bilingual Multimodal Benchmark for Cognitively Informed Evaluation of Vision-Language Models4 Jun 2026
- 2026IVR-R1: Refining Trajectories through Iterative Visual-Grounded Reasoning in Reinforcement Learning18 May 2026
- 2026On the Cultural Anachronism and Temporal Reasoning in Vision Language Models14 May 2026
- 2026HyperGVL: Benchmarking and Improving Large Vision-Language Models in Hypergraph Understanding and Reasoning17 Apr 2026
- 2026Collaborative Multi-Agent Scripts Generation for Enhancing Imperfect-Information Reasoning in Murder Mystery Games13 Apr 2026
- 2026MERGE: Guided Vision-Language Models for Multi-Actor Event Reasoning and Grounding in Human-Robot Interaction19 Mar 2026
- 2026It's Time to Get It Right: Improving Analog Clock Reading and Clock-Hand Spatial Reasoning in Vision-Language Models9 Mar 2026
- 2026Lightweight Visual Reasoning for Socially-Aware Robots4 Mar 2026
- 2026Thinking in Structures: Evaluating Spatial Intelligence in Constraint-Governed Spaces8 Feb 2026
- 2025From Building Blocks to Planning: Multi-Step Spatial Reasoning in LLMs with Reinforcement Learning31 Dec 2025
- 2025Rethinking Fine-Tuning: Unlocking Hidden Capabilities in Vision-Language Models28 Dec 2025
12 newest of 36 All 36 (JSON)
- 2026MARE: Multimodal Alignment and Reinforcement for Explainable Deepfake Detection via Vision-Language Modelscited by 4
- 2026ArchSIBench: Benchmarking the Architectural Spatial Intelligence of Vision-Language Modelscited by 2
- 2025BYO-Eval: Build Your Own Dataset for Fine-Grained Visual Assessment of Multimodal Language Modelscited by 1
- 2025CARMA: Context-Aware Situational Grounding of Human-Robot Group Interactions by Combining Vision-Language Models with Object and Action Recognitioncited by 0
- 2026Spatial Colour Mixing Illusions as a Perception Stress Test for Vision-Language Modelscited by 0
- 2026Revealing Multi-View Hallucination in Large Vision-Language Modelscited by 0
- 2026ExpArt-KG: Artwork Image Description Generation through Iterative Exploration of Knowledge Graphscited by 0
- 2026Benchmarking Vision-Language Models for Automated Pathology Diagnosis and Report Generationcited by 0
- 2026Benchmarking Vision-Language Models for Automated Pathology Diagnosis and Report Generation1 Sep 2026
- 2026ExpArt-KG: Artwork Image Description Generation through Iterative Exploration of Knowledge Graphs1 Sep 2026
- 2026ArchSIBench: Benchmarking the Architectural Spatial Intelligence of Vision-Language Models20 May 2026
- 2026Revealing Multi-View Hallucination in Large Vision-Language Models25 Mar 2026
- 2026Spatial Colour Mixing Illusions as a Perception Stress Test for Vision-Language Models6 Mar 2026
- 2026MARE: Multimodal Alignment and Reinforcement for Explainable Deepfake Detection via Vision-Language Models28 Jan 2026
- 2025CARMA: Context-Aware Situational Grounding of Human-Robot Group Interactions by Combining Vision-Language Models with Object and Action Recognition25 Jun 2025
- 2025BYO-Eval: Build Your Own Dataset for Fine-Grained Visual Assessment of Multimodal Language Models5 Jun 2025
Similar, not cited
Close in wording, no citation either way
- 2024Exploring the Frontier of Vision-Language Models: A Survey of Current Methodologies and Future Directions
- 2023ReForm-Eval: Evaluating Large Vision Language Models via Unified Re-Formulation of Task-Oriented Benchmarks
- 2025A Survey on Efficient Vision-Language Models
- 2023LVLM-eHub: A Comprehensive Evaluation Benchmark for Large Vision-Language Models
- 2025Visual Large Language Models for Generalized and Specialized Applications
- 2024What matters when building vision-language models?
- 2024MM-R3: On (In-)Consistency of Vision-Language Models (VLMs)
- 2024FiVL: A Framework for Improved Vision-Language Alignment through the Lens of Training, Evaluation and Explainability