mlsanityData to 7 Oct 2026

A Comprehensive Survey and Guide to Multimodal Large Language Models in Vision-Language Tasks

arXiv 2411.06284 ↗First posted 9 Nov 2024cs.AIcited by 27

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

This survey and application guide to multimodal large language models(MLLMs) explores the rapidly developing field of MLLMs, examining their architectures, applications, and impact on AI and Generative Models. Starting with foundational concepts, we delve into how MLLMs integrate various data types, including text, images, video and audio, to enable complex AI systems for cross-modal understanding and generation. It covers essential topics such as training methods, architectural components, and practical applications in various fields, from visual storytelling to enhanced accessibility. Through detailed case studies and technical analysis, the text examines prominent MLLM implementations while addressing key challenges in scalability, robustness, and cross-modal learning. Concluding with a discussion of ethical considerations, responsible AI development, and future directions, this authoritative resource provides both theoretical frameworks and practical insights. It offers a balanced perspective on the opportunities and challenges in the development and deployment of MLLMs, and is highly valuable for researchers, practitioners, and students interested in the intersection of natural language processing and computer vision.

Key facts. Cited by 27 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 15 of its 241 references that are in the map. First posted on arXiv 9 Nov 2024.

Builds on

15 of 241 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024Img-Diff: Contrastive Data Synthesis for Multimodal Large Language Models8 Aug
  2. 2023MultiWay-Adapater: Adapting large-scale multi-modal models for scalable image-text retrieval4 Sep
  3. 2023A Survey on Multimodal Large Language Models23 Jun

Other references · 12

  1. 2021Parameter-efficient Multi-task Fine-tuning for Transformers via Shared Hypernetworkscited by 278
  2. 2024Scene-LLM: Extending Language Model for 3D Visual Understanding and Reasoningcited by 170
  3. 2023Hallucination Augmented Contrastive Learning for Multimodal Large Language Modelcited by 119
  4. 2023MFTCoder: Boosting Code LLMs with Multitask Fine-Tuningcited by 34
  5. 2023RAMM: Retrieval-augmented Biomedical Visual Question Answering with Multi-modal Pre-trainingcited by 31
  6. 2023Masked Vision and Language Pre-training with Unimodal and Multimodal Contrastive Losses for Medical Visual Question Answeringcited by 29
  7. 2024OmniDrive: A Holistic Vision-Language Dataset for Autonomous Driving with Counterfactual Reasoningcited by 26
  8. 2023Improving Zero-shot Visual Question Answering via Large Language Models with Reasoning Question Promptscited by 19
  9. 2024M2IST: Multi-Modal Interactive Side-Tuning for Efficient Referring Expression Comprehensioncited by 11
  10. 2022SPT: Semi-Parametric Prompt Tuning for Multitask Prompted Learningcited by 5
  11. 2024Transforming LLMs into Cross-modal and Cross-lingual Retrieval Systemscited by 5
  12. 2024Let Storytelling Tell Vivid Stories: An Expressive and Fluent Multimodal Storytellercited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 17; 2026: 10.

27 papers.

Sort:

By area:

  1. 2025A Survey of Token Compression for Efficient Multimodal Large Language Modelscited by 22
  2. 2025Where MLLMs Attend and What They Rely On: Explaining Autoregressive Token Generationcited by 15
  3. 2025Building Trustworthy Multimodal AI: A Review of Fairness, Transparency, and Ethics in Vision-Language Taskscited by 7
  4. 2025Multi-modal Summarization in Model-Based Engineering: Automotive Software Development Case Studycited by 6
  5. 2026Toward Native Multimodal Modeling: A Roadmapcited by 6
  6. 2025AVA-Bench: Atomic Visual Ability Benchmark for Vision Foundation Modelscited by 5
  7. 2025VF-Eval: Evaluating Multimodal LLMs for Generating Feedback on AIGC Videoscited by 4
  8. 2025CCD: Mitigating Hallucinations in Radiology MLLMs via Clinical Contrastive Decodingcited by 4
  9. 2025Affective Multimodal Agents with Proactive Knowledge Grounding for Emotionally Aligned Marketing Dialoguecited by 4
  10. 2025HUMORCHAIN: Theory-Guided Multi-Stage Reasoning for Interpretable Multimodal Humor Generationcited by 4
  11. 2026Towards Understanding Modality Interaction in Multimodal Language Models via Partial Information Decompositioncited by 4
  12. 2025ClinKD: Cross-Modal Clinical Knowledge Distiller For Multi-Task Medical Imagescited by 2
15 more
  1. 2025Exploring Implicit Visual Misunderstandings in Multimodal Large Language Models through Attention Analysiscited by 2
  2. 2025Falcon: A Cross-Modal Evaluation Dataset for Comprehensive Safety Perceptioncited by 2
  3. 2025LumiGen: An LVLM-Enhanced Iterative Framework for Fine-Grained Text-to-Image Generationcited by 1
  4. 2025Vision-Driven Prompt Optimization for Large Language Models in Multimodal Generative Taskscited by 0
  5. 2025VisuCraft: Enhancing Large Vision-Language Models for Complex Visual-Guided Creative Content Generation via Structured Information Extractioncited by 0
  6. 2025Structuring Reasoning for Complex Rules Beyond Flat Representationscited by 0
  7. 2025When Harmful Content Gets Camouflaged: Unveiling Perception Failure of LVLMs with CamHarmTIcited by 0
  8. 2026LLaVA-CKD: Bottom-Up Cascaded Knowledge Distillation for Vision-Language Modelscited by 0
  9. 2026LatentRouter: Can We Choose the Right Multimodal Model Before Seeing Its Answer?cited by 0
  10. 2026From Structure to Synergy: A Survey of Vision-Language Perception Paradigm Evolution in Multimodal Large Language Modelscited by 0
  11. 2026EVLA: An Electro-Aware Multimodal Assistant for Physically-Grounded Driving Reasoning and Controlcited by 0
  12. 2026DAIN: Dynamic Agent-Based Interaction Network for Efficient and Collaborative Multimodal Reasoningcited by 0
  13. 2026Beyond Skepticism: Evaluating LLMs Pedagogical Intent Reasoning with the Adaptive Pedagogical Vigilance Frameworkcited by 0
  14. 2026From Modalities to Propositions: A Language-Centric Framework for Multimodal Intelligencecited by 0
  15. 2026Rethinking CD: A Reproducibility Study and Extension on the Ineffectiveness of Contrastive Decoding at Mitigating Object Hallucinations in MLLMscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024A Comprehensive Review of Multimodal Large Language Models: Performance and Challenges Across Different Tasks
  2. 2024The Revolution of Multimodal Large Language Models: A Survey
  3. 2024Explainable and Interpretable Multimodal Large Language Models: A Comprehensive Survey
  4. 2024A Survey on Multimodal Benchmarks: In the Era of Large AI Models
  5. 2024Exploring the Frontier of Vision-Language Models: A Survey of Current Methodologies and Future Directions
  6. 2024Surveying the MLLM Landscape: A Meta-Review of Current Surveys
  7. 2025Multimodal Large Language Models for Text-rich Image Understanding: A Comprehensive Review
  8. 2024EE-MLLM: A Data-Efficient and Compute-Efficient Multimodal Large Language Model