mlsanityData to 7 Oct 2026

From Structure to Synergy: A Survey of Vision-Language Perception Paradigm Evolution in Multimodal Large Language Models

arXiv 2606.26196 ↗First posted 24 Jun 2026cs.CL cs.AI cs.CVcited by 0

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Multimodal Large Language Models (MLLMs) have recently made remarkable progress in unifying vision-language understanding and reasoning, especially following the introduction of models such as OpenAI's O-series and DeepSeek's R-series, which have driven a paradigm shift toward perception-centric intelligence. However, there remains a lack of systematic surveys that examine perception from a truly unified vision-language perspective -- one that treats vision and language as an inseparable modality. Existing reviews are often fragmented, focusing separately on either vision or language, and thus rarely capture the cross-modal evolution of perception as an integrated capability. To bridge this gap, we present the first systematic survey of unified vision-language perception in MLLMs. Specifically, we (1) formalize MLLM perception as an intrinsic, unified vision-language capability analogous to human innate perception, (2) introduce a five-stage taxonomy tracing the paradigm evolution of MLLM perception and survey representative methods and milestones at each phase, and (3) identify open challenges and outline promising research directions toward truly general, unified multimodal intelligence. We hope our study will provide both a foundational understanding and an actionable roadmap to foster further innovation on the path toward artificial general intelligence (AGI).

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 204 of its 228 references that are in the map. First posted on arXiv 24 Jun 2026.

Builds on

204 of 228 references.

Closest ancestors · 35

Same direction, newest first

  1. 2026Modality Gap-Driven Subspace Alignment Training Paradigm For Multimodal Large Language Models2 Feb
  2. 2025MAC: A Live Benchmark for Multimodal Large Language Models in Scientific Understanding14 Aug
  3. 2025Zoom-Refine: Boosting High-Resolution Multimodal Understanding via Localized Zoom and Self-Refinement2 Jun
  4. 2025Information Density Principle for MLLM Benchmarks13 Mar
  5. 2025Introducing Visual Perception Token into Multimodal Large Language Model24 Feb
  6. 2025PerPO: Perceptual Preference Optimization via Discriminative Rewarding5 Feb
  7. 2024Task Preference Optimization: Improving Multimodal Large Language Models with Vision Task Alignment26 Dec
  8. 2024ChatRex: Taming Multimodal LLM for Joint Perception and Understanding27 Nov
  9. 2024ZoomEye: Enhancing Multimodal LLMs with Human-Like Zooming Capabilities through Tree-Based Image Exploration25 Nov
  10. 2024A Comprehensive Survey and Guide to Multimodal Large Language Models in Vision-Language Tasks9 Nov
  11. 2024LLaVA-Critic: Learning to Evaluate Multimodal Models3 Oct
  12. 2024Instruction-guided Multi-Granularity Segmentation and Captioning with Large Multimodal Model20 Sep
23 more
  1. 2024Eagle: Exploring The Design Space for Multimodal LLMs with Mixture of Encoders28 Aug
  2. 2024MoME: Mixture of Multimodal Experts for Generalist Multimodal Large Language Models17 Jul
  3. 2024MG-LLaVA: Towards Multi-Granularity Visual Instruction Tuning25 Jun
  4. 2024VisionLLM v2: An End-to-End Generalist Multimodal Large Language Model for Hundreds of Vision-Language Tasks12 Jun
  5. 2024TextCoT: Zoom In for Enhanced Multimodal Text-Rich Image Understanding15 Apr
  6. 2024Draw-and-Understand: Leveraging Visual Prompts to Enable MLLMs to Comprehend What You Want29 Mar
  7. 2024Lumen: Unleashing Versatile Vision-Centric Capabilities of Large Multimodal Models12 Mar
  8. 2024Multi-modal Instruction Tuned LLMs with Fine-grained Visual Perception5 Mar
  9. 2024DualFocus: Integrating Macro and Micro Perspectives in Multi-modal Large Language Models22 Feb
  10. 2024The Revolution of Multimodal Large Language Models: A Survey19 Feb
  11. 2024SPHINX-X: Scaling Data and Parameters for a Family of Multi-modal Large Language Models8 Feb
  12. 2024MM-LLMs: Recent Advances in MultiModal Large Language Models24 Jan
  13. 2023VCoder: Versatile Vision Encoders for Multimodal Large Language Models21 Dec
  14. 2023Osprey: Pixel Understanding with Visual Instruction Tuning15 Dec
  15. 2023Honeybee: Locality-enhanced Projector for Multimodal LLM11 Dec
  16. 2023ViP-LLaVA: Making Large Multimodal Models Understand Arbitrary Visual Prompts1 Dec
  17. 2023LION : Empowering Multimodal Large Language Model with Dual-Level Visual Knowledge20 Nov
  18. 2023LLaVA-Plus: Learning to Use Tools for Creating Multimodal Agents9 Nov
  19. 2023u-LLaVA: Unifying Multi-Modal Tasks via Large Language Model9 Nov
  20. 2023Pink: Unveiling the Power of Referential Comprehension for Multi-modal LLMs1 Oct
  21. 2023Investigating the Catastrophic Forgetting in Multimodal Large Language Models19 Sep
  22. 2023BuboGPT: Enabling Visual Grounding in Multi-Modal LLMs17 Jul
  23. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul

Other references · 169

  1. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  5. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  6. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  7. 2023Segment Anythingcited by 8,158
  8. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  9. 2023Visual Instruction Tuningcited by 7,792
  10. 2017Mask R-CNNcited by 7,259
  11. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,410
  12. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
38 more
  1. 2024SAM 2: Segment Anything in Images and Videoscited by 2,784
  2. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,678
  3. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  4. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,449
  5. 2022DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detectioncited by 1,306
  6. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
  7. 2016Modeling Context in Referring Expressionscited by 1,054
  8. 2023LISA: Reasoning Segmentation via Large Language Modelcited by 770
  9. 2024Emu3: Next-Token Prediction is All You Needcited by 761
  10. 2017Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithmcited by 723
  11. 2023BridgeData V2: A Dataset for Robot Learning at Scalecited by 634
  12. 2023ViperGPT: Visual Inference via Python Execution for Reasoningcited by 591
  13. 2025VLM-R1: A Stable and Generalizable R1-style Large Vision-Language Modelcited by 515
  14. 2022Visual Programming: Compositional visual reasoning without trainingcited by 507
  15. 2025Visual-RFT: Visual Reinforcement Fine-Tuningcited by 501
  16. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularitycited by 498
  17. 2023V*: Guided Visual Search as a Core Mechanism in Multimodal LLMscited by 459
  18. 2023GLaMM: Pixel Grounding Large Multimodal Modelcited by 447
  19. 2023TimeChat: A Time-sensitive Multimodal Large Language Model for Long Video Understandingcited by 399
  20. 2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 381
  21. 2024The Platonic Representation Hypothesiscited by 362
  22. 2023Scaling Open-Vocabulary Object Detectioncited by 335
  23. 2023Recognize Anything: A Strong Image Tagging Modelcited by 305
  24. 2023GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interestcited by 300
  25. 2024SpatialRGPT: Grounded Spatial Reasoning in Vision Language Modelscited by 297
  26. 2025Pixel Reasoner: Incentivizing Pixel-Space Reasoning with Curiosity-Driven Reinforcement Learningcited by 267
  27. 2024Visual Sketchpad: Sketching as a Visual Chain of Thought for Multimodal Language Modelscited by 260
  28. 2023VTimeLLM: Empower LLM to Grasp Video Momentscited by 253
  29. 2025MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attentioncited by 248
  30. 2025Seg-Zero: Reasoning-Chain Guided Segmentation via Cognitive Reinforcementcited by 234
  31. 2023PixelLM: Pixel Reasoning with Large Multimodal Modelcited by 220
  32. 2025Unified Reward Model for Multimodal Understanding and Generationcited by 194
  33. 2024LLaVA-UHD: an LMM Perceiving Any Aspect Ratio and High-Resolution Imagescited by 193
  34. 2025Multimodal Chain-of-Thought Reasoning: A Comprehensive Surveycited by 176
  35. 2025OpenThinkIMG: Learning to Think with Images via Visual Tool Reinforcement Learningcited by 163
  36. 2025Sa2VA: Marrying SAM2 with MLLM for Dense Grounded Understanding of Images and Videoscited by 154
  37. 2023GSVA: Generalized Segmentation via Multimodal Large Language Modelscited by 153
  38. 2024Tarsier: Recipes for Training and Evaluating Large Video Description Modelscited by 140

50 most cited of 169 All 169 (JSON)

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Vision Language Models: A Survey of 26K Papers
  2. 2026PerceptionBench: Evaluating Atomic Visual Perception in Multimodal Large Language Models
  3. 2022Vision-Language Intelligence: Tasks, Representation Learning, and Large Models
  4. 2024Evaluating and Advancing Multimodal Large Language Models in Perception Ability Lens
  5. 2024Exploring the Frontier of Vision-Language Models: A Survey of Current Methodologies and Future Directions
  6. 2026The Last Visible Pixel: Probing Fine-Scale Perception in Vision-Language Models
  7. 2025Multimodal LLM Augmented Reasoning for Interpretable Visual Perception Analysis
  8. 2026Almieyar-Oryx-BloomBench: A Bilingual Multimodal Benchmark for Cognitively Informed Evaluation of Vision-Language Models