mlsanityData to 7 Oct 2026

A Survey of State of the Art Large Vision Language Models: Alignment, Benchmark, Evaluations and Challenges

arXiv 2501.02189 ↗First posted 4 Jan 2025cs.CV cs.AI cs.CLcited by 46

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

also in 2 more
Abstract (arXiv)

Large vision-language models (VLMs) have evolved rapidly from contrastive image-text encoders and adapter-based assistants into natively multimodal foundation models that support long-context reasoning, agentic workflows, and, increasingly, unified perception-generation-action loops. Since 2025, the frontier has consolidated around a small number of large pretraining families, including GPT, Gemini, Claude, Grok, Qwen, Gemma, DeepSeek, Kimi, and MiniMax, while evaluation has shifted from short-form visual question answering toward spatial, temporal, embodied, and calibration-sensitive benchmarks. In this survey, we provide an updated overview of VLMs through 2026 with an emphasis on three developments. First, we trace the architectural evolution from contrastive or bridged two-tower models to LLM-backbone adapter models, native multimodal-input models, omni-modal unified input/output models, and emerging world-action models. Second, we summarize how benchmarks expanded from classical OCR, VQA, and chart understanding toward long-video reasoning, embodied evaluation, reward-model judging, and domain-specific decision making. Third, we review the growing role of post-training, including supervised fine-tuning, preference optimization, and reinforcement learning methods such as GRPO-style multimodal alignment. Rather than exhaustively listing every release, we focus on representative frontier families, influential benchmarks, and the major open challenges that remain in hallucination, safety, efficiency, data quality, and multimodal alignment.

Key facts. Cited by 46 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 145 of its 226 references that are in the map. First posted on arXiv 4 Jan 2025.

Builds on

145 of 226 references.

Closest ancestors · 9

Same direction, newest first

  1. 2024NaturalBench: Evaluating Vision-Language Models on Natural Adversarial Samples18 Oct
  2. 2024Benchmarking Vision Language Models for Cultural Understanding15 Jul
  3. 2024An Introduction to Vision-Language Modeling27 May
  4. 2024Enhancing Visual-Language Modality Alignment in Large Vision Language Models via Self-Improvement24 May
  5. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?29 Mar
  6. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond24 Aug
  7. 2022Flamingo: a Visual Language Model for Few-Shot Learning29 Apr
  8. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation28 Jan
  9. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervision24 Aug

Other references · 136

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  7. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  8. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  9. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  10. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  11. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  12. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
38 more
  1. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  2. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  3. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  4. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,388
  5. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  6. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  7. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
  8. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 1,908
  9. 2023VBench: Comprehensive Benchmark Suite for Video Generative Modelscited by 1,755
  10. 2019Towards VQA Models That Can Readcited by 1,696
  11. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  12. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
  13. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,449
  14. 2018World Modelscited by 1,444
  15. 2016Context Encoders: Feature Learning by Inpaintingcited by 1,424
  16. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,375
  17. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,288
  18. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  19. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
  20. 2020Leveraging Passage Retrieval with Generative Models for Open Domain Question Answeringcited by 1,203
  21. 2019Habitat: A Platform for Embodied AI Researchcited by 1,163
  22. 2023GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignmentcited by 946
  23. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 929
  24. 2024WorldSimBench: Towards Video Generation Models as World Simulatorscited by 867
  25. 2017AI2-THOR: An Interactive 3D Environment for Visual AIcited by 850
  26. 2019Self-Supervised Learning of Pretext-Invariant Representationscited by 806
  27. 2016A Diagram Is Worth A Dozen Imagescited by 793
  28. 2017Reading Wikipedia to Answer Open-Domain Questionscited by 777
  29. 2023AGIEval: A Human-Centric Benchmark for Evaluating Foundation Modelscited by 701
  30. 2023C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Modelscited by 694
  31. 2023CogAgent: A Visual Language Model for GUI Agentscited by 692
  32. 2023VILA: On Pre-training for Visual Language Modelscited by 686
  33. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)cited by 677
  34. 2023EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understandingcited by 666
  35. 2018Object Hallucination in Image Captioningcited by 661
  36. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 657
  37. 2023Bias and Fairness in Large Language Models: A Surveycited by 626
  38. 2023GAIA-1: A Generative World Model for Autonomous Drivingcited by 618

50 most cited of 136 All 136 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 29; 2026: 17.

46 papers.

Sort:

By area:

  1. 2025Thinking with Images for Multimodal Reasoning: Foundations, Methods, and Future Frontierscited by 193
  2. 2026Graph-of-Skills: Dependency-Aware Structural Retrieval for Massive Agent Skillscited by 53
  3. 2025ColorBench: Can VLMs See and Understand the Colorful World? A Comprehensive Benchmark for Color Perception, Reasoning, and Robustnesscited by 28
  4. 2025DreamPRM: Domain-Reweighted Process Reward Model for Multimodal Reasoningcited by 15
  5. 2025SpinBench: Perspective and Rotation as a Lens on Spatial Reasoning in VLMscited by 12
  6. 2025VACoT: Rethinking Visual Data Augmentation with VLMscited by 8
  7. 2025CausalVLBench: Benchmarking Visual Causal Reasoning in Large Vision-Language Modelscited by 7
  8. 2025ORIGAMISPACE: Benchmarking Multimodal LLMs in Multi-Step Spatial Reasoning with Mathematical Constraintscited by 7
  9. 2025Small Drafts, Big Verdict: Information-Intensive Visual Reasoning via Speculationcited by 6
  10. 2026Collaborative Multi-Agent Scripts Generation for Enhancing Imperfect-Information Reasoning in Murder Mystery Gamescited by 6
  11. 2025SemVink: Advancing VLMs' Semantic Understanding of Optical Illusions via Visual Global Thinkingcited by 5
  12. 2025MASS: Motion-Aware Spatial-Temporal Grounding for Physics Reasoning and Comprehension in Vision-Language Modelscited by 5
34 more
  1. 2025Limits and Gains of Test-Time Scaling in Vision-Language Reasoningcited by 5
  2. 2025HUMORCHAIN: Theory-Guided Multi-Stage Reasoning for Interpretable Multimodal Humor Generationcited by 4
  3. 2026MARE: Multimodal Alignment and Reinforcement for Explainable Deepfake Detection via Vision-Language Modelscited by 4
  4. 2025VisChainBench: A Benchmark for Multi-Turn, Multi-Image Visual Reasoning Beyond Language Priorscited by 3
  5. 2025Modeling Beyond MOS: Quality Assessment Models Must Integrate Context, Reasoning, and Multimodalitycited by 2
  6. 2025FC-MIR: A Mobile Screen Awareness Framework for Intent-Aware Recommendation based on Frame-Compressed Multimodal Trajectory Reasoningcited by 2
  7. 2026Thinking in Structures: Evaluating Spatial Intelligence in Constraint-Governed Spacescited by 2
  8. 2026ArchSIBench: Benchmarking the Architectural Spatial Intelligence of Vision-Language Modelscited by 2
  9. 2025BYO-Eval: Build Your Own Dataset for Fine-Grained Visual Assessment of Multimodal Language Modelscited by 1
  10. 2025COREVQA: A Crowd Observation and Reasoning Entailment Visual Question Answering Benchmarkcited by 1
  11. 2025FOUNDv2: Learning Unified User Quantized Tokenizers for User Representationcited by 1
  12. 2025Understanding Space Is Rocket Science -- Only Top Reasoning Models Can Solve Spatial Understanding Taskscited by 1
  13. 2025AgentCaster: Reasoning-Guided Tornado Forecastingcited by 1
  14. 2025COVLM-RL: Critical Object-Oriented Reasoning for Autonomous Driving Using VLM-Guided Reinforcement Learningcited by 1
  15. 2025From Building Blocks to Planning: Multi-Step Spatial Reasoning in LLMs with Reinforcement Learningcited by 1
  16. 2026It's Time to Get It Right: Improving Analog Clock Reading and Clock-Hand Spatial Reasoning in Vision-Language Modelscited by 1
  17. 2026Almieyar-Oryx-BloomBench: A Bilingual Multimodal Benchmark for Cognitively Informed Evaluation of Vision-Language Modelscited by 1
  18. 2026MIRROR: Learning from the Other View for Multi-Modal Reasoningcited by 1
  19. 2025Elevating Visual Question Answering through Implicitly Learned Reasoning Pathways in LVLMscited by 0
  20. 2025CARMA: Context-Aware Situational Grounding of Human-Robot Group Interactions by Combining Vision-Language Models with Object and Action Recognitioncited by 0
  21. 2025Ascending the Infinite Ladder: Benchmarking Spatial Deformation Reasoning in Vision-Language Modelscited by 0
  22. 2025Decomposing Visual Classification: Assessing Tree-Based Reasoning in VLMscited by 0
  23. 2025Rule-Based Reinforcement Learning for Document Image Classification with Vision Language Modelscited by 0
  24. 2025AgroCoT: A Chain-of-Thought Benchmark for Evaluating Reasoning in Vision-Language Models for Agriculturecited by 0
  25. 2025Rethinking Fine-Tuning: Unlocking Hidden Capabilities in Vision-Language Modelscited by 0
  26. 2026Lightweight Visual Reasoning for Socially-Aware Robotscited by 0
  27. 2026Spatial Colour Mixing Illusions as a Perception Stress Test for Vision-Language Modelscited by 0
  28. 2026MERGE: Guided Vision-Language Models for Multi-Actor Event Reasoning and Grounding in Human-Robot Interactioncited by 0
  29. 2026Revealing Multi-View Hallucination in Large Vision-Language Modelscited by 0
  30. 2026HyperGVL: Benchmarking and Improving Large Vision-Language Models in Hypergraph Understanding and Reasoningcited by 0
  31. 2026On the Cultural Anachronism and Temporal Reasoning in Vision Language Modelscited by 0
  32. 2026IVR-R1: Refining Trajectories through Iterative Visual-Grounded Reasoning in Reinforcement Learningcited by 0
  33. 2026ExpArt-KG: Artwork Image Description Generation through Iterative Exploration of Knowledge Graphscited by 0
  34. 2026Benchmarking Vision-Language Models for Automated Pathology Diagnosis and Report Generationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Exploring the Frontier of Vision-Language Models: A Survey of Current Methodologies and Future Directions
  2. 2023ReForm-Eval: Evaluating Large Vision Language Models via Unified Re-Formulation of Task-Oriented Benchmarks
  3. 2025A Survey on Efficient Vision-Language Models
  4. 2023LVLM-eHub: A Comprehensive Evaluation Benchmark for Large Vision-Language Models
  5. 2025Visual Large Language Models for Generalized and Specialized Applications
  6. 2024What matters when building vision-language models?
  7. 2024MM-R3: On (In-)Consistency of Vision-Language Models (VLMs)
  8. 2024FiVL: A Framework for Improved Vision-Language Alignment through the Lens of Training, Evaluation and Explainability