mlsanityData to 7 Oct 2026

A Comprehensive Review of Multimodal Large Language Models: Performance and Challenges Across Different Tasks

arXiv 2408.01319 ↗First posted 2 Aug 2024cs.AIcited by 57

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

In an era defined by the explosive growth of data and rapid technological advancements, Multimodal Large Language Models (MLLMs) stand at the forefront of artificial intelligence (AI) systems. Designed to seamlessly integrate diverse data types-including text, images, videos, audio, and physiological sequences-MLLMs address the complexities of real-world applications far beyond the capabilities of single-modality systems. In this paper, we systematically sort out the applications of MLLM in multimodal tasks such as natural language, vision, and audio. We also provide a comparative analysis of the focus of different MLLMs in the tasks, and provide insights into the shortcomings of current MLLMs, and suggest potential directions for future research. Through these discussions, this paper hopes to provide valuable insights for the further development and application of MLLM.

Key facts. Cited by 57 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 90 of its 114 references that are in the map. First posted on arXiv 2 Aug 2024.

Builds on

90 of 114 references.

Closest ancestors · 15

Same direction, newest first

  1. 2024MANTIS: Interleaved Multi-Image Instruction Tuning2 May
  2. 2024AnyGPT: Unified Multimodal LLM with Discrete Sequence Modeling19 Feb
  3. 2024LLaVA-MoLE: Sparse Mixture of LoRA Experts for Mitigating Data Conflicts in Instruction Finetuning MLLMs29 Jan
  4. 2024MM-LLMs: Recent Advances in MultiModal Large Language Models24 Jan
  5. 2023VILA: On Pre-training for Visual Language Models12 Dec
  6. 2023Multimodal Large Language Models: A Survey22 Nov
  7. 2023Improved Baselines with Visual Instruction Tuning5 Oct
  8. 2023NExT-GPT: Any-to-Any Multimodal LLM11 Sep
  9. 2023BuboGPT: Enabling Visual Grounding in Multi-Modal LLMs17 Jul
  10. 2023A Survey on Multimodal Large Language Models23 Jun
  11. 2023Macaw-LLM: Multi-Modal Language Modeling with Image, Audio, Video, and Text Integration15 Jun
  12. 2023Generating Images with Multimodal Language Models26 May
3 more
  1. 2023SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilities18 May
  2. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning11 May
  3. 2023VPGTrans: Transfer Visual Prompt Generator across LLMs2 May

Other references · 75

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2023GPT-4 Technical Reportcited by 18,132
  5. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  7. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  8. 2023Visual Instruction Tuningcited by 7,792
  9. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  10. 2022Generative Adversarial Networkscited by 5,603
  11. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  12. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
38 more
  1. 2022Video Diffusion Modelscited by 2,280
  2. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  3. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,461
  4. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,385
  5. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  6. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,175
  7. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,173
  8. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,162
  9. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,086
  10. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,047
  11. 2023VideoChat: Chat-Centric Video Understandingcited by 851
  12. 2021SoundStream: An End-to-End Neural Audio Codeccited by 756
  13. 2024VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMscited by 749
  14. 2021NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actionscited by 735
  15. 2024Yi: Open Foundation Models by 01.AIcited by 706
  16. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Modelscited by 659
  17. 2021Vector-quantized Image Modeling with Improved VQGANcited by 656
  18. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 617
  19. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  20. 2023Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Modelscited by 591
  21. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 560
  22. 2023LLaMA-VID: An Image is Worth 2 Tokens in Large Language Modelscited by 539
  23. 2023SALMONN: Towards Generic Hearing Abilities for Large Language Modelscited by 492
  24. 2024Sora: A Review on Background, Technology, Limitations, and Opportunities of Large Vision Modelscited by 467
  25. 2022BEATs: Audio Pre-Training with Acoustic Tokenizerscited by 378
  26. 2024Mini-Gemini: Mining the Potential of Multi-modality Vision Language Modelscited by 361
  27. 2023AudioPaLM: A Large Language Model That Can Speak and Listencited by 333
  28. 2023PandaGPT: One Model To Instruction-Follow Them Allcited by 330
  29. 2024What matters when building vision-language models?cited by 321
  30. 2021Video Transformer Networkcited by 271
  31. 2023AudioGPT: Understanding and Generating Speech, Music, Sound, and Talking Headcited by 264
  32. 2023MIMIC-IT: Multi-Modal In-Context Instruction Tuningcited by 264
  33. 2023MultiModal-GPT: A Vision and Language Model for Dialogue with Humanscited by 255
  34. 2020DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representationscited by 249
  35. 2024Playground v2.5: Three Insights towards Enhancing Aesthetic Quality in Text-to-Image Generationcited by 249
  36. 2023Multimodal Foundation Models: From Specialists to General-Purpose Assistantscited by 247
  37. 2024EMO: Emote Portrait Alive -- Generating Expressive Portrait Videos with Audio2Video Diffusion Model under Weak Conditionscited by 233
  38. 2023Valley: Video Assistant with Large Language model Enhanced abilitYcited by 231

50 most cited of 75 All 75 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 9; 2025: 31; 2026: 17.

57 papers.

Sort:

By area:

  1. 2024EarthDial: Turning Multi-sensory Earth Observations to Interactive Dialoguescited by 81
  2. 2025Video-Bench: Human-Aligned Video Generation Benchmarkcited by 53
  3. 2024Explainable and Interpretable Multimodal Large Language Models: A Comprehensive Surveycited by 52
  4. 2025Distraction is All You Need for Multimodal Large Language Model Jailbreakingcited by 31
  5. 2025PointArena: Probing Multimodal Grounding Through Language-Guided Pointingcited by 30
  6. 2024Opportunities and Challenges of Large Language Models for Low-Resource Languages in Humanities Researchcited by 26
  7. 2024From Specific-MLLMs to Omni-MLLMs: A Survey on MLLMs Aligned with Multi-modalitiescited by 22
  8. 2025MRAMG-Bench: A Comprehensive Benchmark for Advancing Multimodal Retrieval-Augmented Multimodal Generationcited by 15
  9. 2024Do Language Models Understand Time?cited by 14
  10. 2025Object Detection with Multimodal Large Vision-Language Models: An In-depth Reviewcited by 14
  11. 2024Transcending Language Boundaries: Harnessing LLMs for Low-Resource Language Translationcited by 12
  12. 2025Pushing the Limits of Safety: A Technical Report on the ATLAS Challenge 2025cited by 11
38 more
  1. 2025Position: Foundation Models Need Digital Twin Representationscited by 10
  2. 2025AdaRD-key: Adaptive Relevance-Diversity Keyframe Sampling for Long-form Video understandingcited by 9
  3. 2025Evaluating Mathematical Reasoning Across Large Language Models: A Fine-Grained Approachcited by 8
  4. 2026Thinking with Deltas: Incentivizing Reinforcement Learning via Differential Visual Reasoning Policycited by 6
  5. 2025From large language models to multimodal AI: A scoping review on the potential of generative AI in medicinecited by 5
  6. 2025Can LLMs Generate Behaviors for Embodied Virtual Agents Based on Personality Traits?cited by 5
  7. 2025Beyond Spurious Signals: Debiasing Multimodal Large Language Models via Counterfactual Inference and Adaptive Expert Routingcited by 5
  8. 2024Legal Evalutions and Challenges of Large Language Modelscited by 4
  9. 2025Who Owns the Output? Bridging Law and Technology in LLMs Attributioncited by 4
  10. 2025Text2midi-InferAlign: Improving Symbolic Music Generation with Inference-Time Alignmentcited by 4
  11. 2025CCD: Mitigating Hallucinations in Radiology MLLMs via Clinical Contrastive Decodingcited by 4
  12. 2026MTMCS-Bench: Evaluating Contextual Safety of Multimodal Large Language Models in Multi-Turn Dialoguescited by 4
  13. 2026Glance-or-Gaze: Incentivizing LMMs to Adaptively Focus Search via Reinforcement Learningcited by 4
  14. 2025Large Language Models for Bioinformaticscited by 3
  15. 2025DataPuzzle: Breaking Free from the Hallucinated Promise of LLMs in Data Analysiscited by 3
  16. 2025Multimodal Representation Learning and Fusioncited by 3
  17. 2025Multi-Amateur Contrastive Decoding for Text Generationcited by 3
  18. 2025Large Foundation Models for Trajectory Prediction in Autonomous Driving: A Comprehensive Surveycited by 3
  19. 2025EduVidQA: Generating and Evaluating Long-form Answers to Student Questions based on Lecture Videoscited by 3
  20. 2025Q-MLLM: Vector Quantization for Robust Multimodal Large Language Model Securitycited by 2
  21. 2026Multi-turn Jailbreaking Attack in Multi-Modal Large Language Modelscited by 2
  22. 2026Privacy in Human-AI Romantic Relationships: Concerns, Boundaries, and Agencycited by 2
  23. 2026From 3D Pose to Prose: Biomechanics-Grounded Vision--Language Coachingcited by 2
  24. 2026MCERF: Advancing Multimodal LLM Evaluation of Engineering Documentation with Enhanced Retrievalcited by 2
  25. 2025MLLM-based Speech Recognition: When and How is Multimodality Beneficial?cited by 1
  26. 2025Learning to Hear by Seeing: It's Time for Vision Language Models to Understand Artistic Emotion from Sight and Soundcited by 1
  27. 2025R-AVST: Empowering Video-LLMs with Fine-Grained Spatio-Temporal Reasoning in Complex Audio-Visual Scenarioscited by 1
  28. 2025FPBench: A Comprehensive Benchmark of Multimodal Large Language Models for Fingerprint Analysiscited by 1
  29. 2026LLM-Assisted Logic Rule Learning: Scaling Human Expertise for Time Series Anomaly Detectioncited by 1
  30. 2026MM-BizRAG: Rethinking Multimodal Retrieval-Augmented Generation for General Purpose Enterprise Q&Acited by 1
  31. 2024Recent advances in deep learning and language models for studying the microbiomecited by 0
  32. 2024Visual Error Patterns in Multi-Modal AI: A Statistical Approachcited by 0
  33. 2025MindMem: Multimodal for Predicting Advertisement Memorability Using LLMs and Deep Learningcited by 0
  34. 2025ORACLE-Grasp: Zero-Shot Affordance-Aligned Robotic Grasping using Large Multimodal Modelscited by 0
  35. 2025Fusion to Enhance: Fusion Visual Encoder to Enhance Multimodal Language Modelcited by 0
  36. 2025LMM-IQA: Image Quality Assessment for Low-Dose CT Imagingcited by 0
  37. 2025DialogGraph-LLM: Graph-Informed LLMs for End-to-End Audio Dialogue Intent Recognitioncited by 0
  38. 2026Mimic Human Cognition, Master Multi-Image Reasoning: A Meta-Action Framework for Enhanced Visual Understandingcited by 0

50 most cited of 57 All 57 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024A Comprehensive Survey and Guide to Multimodal Large Language Models in Vision-Language Tasks
  2. 2024A Survey of Multimodal Large Language Model from A Data-centric Perspective
  3. 2024Surveying the MLLM Landscape: A Meta-Review of Current Surveys
  4. 2024A Survey on Multimodal Benchmarks: In the Era of Large AI Models
  5. 2024The Revolution of Multimodal Large Language Models: A Survey
  6. 2024A Survey on Evaluation of Multimodal Large Language Models
  7. 2024Efficient Multimodal Large Language Models: A Survey
  8. 2025Empowering Multimodal LLMs with External Tools: A Comprehensive Survey