mlsanityData to 14 Oct 2026

mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaboration

arXiv 2311.04257 ↗First posted 7 Nov 2023cs.CL cs.CVcited by 547

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Multi-modal Large Language Models (MLLMs) have demonstrated impressive instruction abilities across various open-ended tasks. However, previous methods primarily focus on enhancing multi-modal capabilities. In this work, we introduce a versatile multi-modal large language model, mPLUG-Owl2, which effectively leverages modality collaboration to improve performance in both text and multi-modal tasks. mPLUG-Owl2 utilizes a modularized network design, with the language decoder acting as a universal interface for managing different modalities. Specifically, mPLUG-Owl2 incorporates shared functional modules to facilitate modality collaboration and introduces a modality-adaptive module that preserves modality-specific features. Extensive experiments reveal that mPLUG-Owl2 is capable of generalizing both text tasks and multi-modal tasks and achieving state-of-the-art performances with a single generic model. Notably, mPLUG-Owl2 is the first MLLM model that demonstrates the modality collaboration phenomenon in both pure-text and multi-modal scenarios, setting a pioneering path in the development of future multi-modal foundation models.

Key facts. Cited by 547 papers in the map (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 60 of its 78 references that are in the map. First posted on arXiv 7 Nov 2023.

Builds on

60 of 78 references.

Closest ancestors · 9

Same direction, newest first

  1. 2023Improved Baselines with Visual Instruction Tuning5 Oct
  2. 2023Aligning Large Multimodal Models with Factually Augmented RLHF25 Sep
  3. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities4 Aug
  4. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension30 Jul
  5. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul
  6. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic27 Jun
  7. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models23 Jun
  8. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning11 May
  9. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuning5 May

Other references · 51

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2023GPT-4 Technical Reportcited by 18,396
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  6. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  7. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  8. 2023Visual Instruction Tuningcited by 7,869
  9. 2020End-to-End Object Detection with Transformerscited by 7,414
  10. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  11. 2016Layer Normalizationcited by 5,669
  12. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
38 more
  1. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  2. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  3. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  4. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
  5. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  6. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,071
  7. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  8. 2017Decoupled Weight Decay Regularizationcited by 1,831
  9. 2020GLU Variants Improve Transformercited by 1,804
  10. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,729
  11. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,471
  12. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,216
  13. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,125
  14. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,121
  15. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,089
  16. 2016Modeling Context in Referring Expressionscited by 1,063
  17. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,062
  18. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,040
  19. 2023VideoChat: Chat-Centric Video Understandingcited by 857
  20. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 766
  21. 2023AGIEval: A Human-Centric Benchmark for Evaluating Foundation Modelscited by 716
  22. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 710
  23. 2023DataComp: In search of the next generation of multimodal datasetscited by 644
  24. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  25. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 565
  26. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  27. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 537
  28. 2022Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Taskscited by 421
  29. 2020TextCaps: a Dataset for Image Captioning with Reading Comprehensioncited by 400
  30. 2017TGIF-QA: Toward Spatio-Temporal Reasoning in Visual Question Answeringcited by 378
  31. 2023MultiModal-GPT: A Vision and Language Model for Dialogue with Humanscited by 255
  32. 2023Q-Bench: A Benchmark for General-Purpose Foundation Models on Low-level Visioncited by 249
  33. 2022Zero-Shot Video Question Answering via Frozen Bidirectional Language Modelscited by 229
  34. 2023mPLUG-DocOwl: Modularized Multimodal Large Language Model for Document Understandingcited by 168
  35. 2023SVIT: Scaling up Visual Instruction Tuningcited by 162
  36. 2023mPLUG-2: A Modularized Multi-modal Foundation Model Across Text, Image and Videocited by 154
  37. 2022HiTeA: Hierarchical Temporal-Aware Video-Language Pre-trainingcited by 72
  38. 2022Masked Vision and Language Modeling for Multi-modal Representation Learningcited by 67

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 19; 2024: 261; 2025: 206; 2026: 61.

547 papers.

Sort:

By area:

  1. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  2. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
  3. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,834
  4. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,599
  5. 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,510
  6. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,050
  7. 2024Emu3: Next-Token Prediction is All You Needcited by 781
  8. 2024VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMscited by 771
  9. 2024Show-o: One Single Transformer to Unify Multimodal Understanding and Generationcited by 752
  10. 2024LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understandingcited by 665
  11. 2024MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?cited by 586
  12. 2024MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmarkcited by 512
38 more
  1. 2023Q-Align: Teaching LMMs for Visual Scoring via Discrete Text-Defined Levelscited by 503
  2. 2024VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Modelscited by 455
  3. 2025DeepEyes: Incentivizing "Thinking with Images" via Reinforcement Learningcited by 410
  4. 2024Hallucination of Multimodal Large Language Models: A Surveycited by 388
  5. 2023Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Modelscited by 357
  6. 2023Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Actioncited by 302
  7. 2023AMBER: An LLM-free Multi-dimensional Benchmark for MLLMs Hallucination Evaluationcited by 300
  8. 2024Mobile-Agent: Autonomous Multi-Modal Mobile Device Agent with Visual Perceptioncited by 293
  9. 2024mPLUG-Owl3: Towards Long Image-Sequence Understanding in Multi-Modal Large Language Modelscited by 291
  10. 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 279
  11. 2024MM-LLMs: Recent Advances in MultiModal Large Language Modelscited by 278
  12. 2025MMaDA: Multimodal Large Diffusion Language Modelscited by 250
  13. 2025LMM-R1: Empowering 3B LMMs with Strong Reasoning Abilities Through Two-Stage Rule-Based RLcited by 248
  14. 2024mPLUG-DocOwl 1.5: Unified Structure Learning for OCR-free Document Understandingcited by 220
  15. 2024LLaVA-UHD: an LMM Perceiving Any Aspect Ratio and High-Resolution Imagescited by 195
  16. 2024Mobile-Agent-v2: Mobile Device Operation Assistant with Effective Navigation via Multi-Agent Collaborationcited by 189
  17. 2024MMBench-Video: A Long-Form Multi-Shot Benchmark for Holistic Video Understandingcited by 182
  18. 2024HALC: Object Hallucination Reduction via Adaptive Focal-Contrast Decodingcited by 174
  19. 2024TextMonkey: An OCR-Free Large Multimodal Model for Understanding Documentcited by 174
  20. 2024Safety Fine-Tuning at (Almost) No Cost: A Baseline for Vision Large Language Modelscited by 167
  21. 2025LLaVA-Mini: Efficient Image and Video Large Multimodal Models with One Vision Tokencited by 166
  22. 2023Compositional Chain-of-Thought Prompting for Large Multimodal Modelscited by 164
  23. 2023Q-Instruct: Improving Low-level Visual Abilities for Multi-modality Foundation Modelscited by 152
  24. 2024SEED-Bench-2-Plus: Benchmarking Multimodal Large Language Models with Text-Rich Visual Comprehensioncited by 146
  25. 2025Teaching Large Language Models to Regress Accurate Image Quality Scores using Score Distributioncited by 141
  26. 2024The Revolution of Multimodal Large Language Models: A Surveycited by 137
  27. 2024Jailbreaking Attack against Multimodal Large Language Modelcited by 135
  28. 2025Dispider: Enabling Video LLMs with Active Real-Time Interaction via Disentangled Perception, Decision, and Reactioncited by 127
  29. 2024Prometheus-Vision: Vision-Language Model as a Judge for Fine-Grained Evaluationcited by 125
  30. 2024AdaShield: Safeguarding Multimodal Large Language Models from Structure-based Attack via Adaptive Shield Promptingcited by 125
  31. 2024Eyes Closed, Safety On: Protecting Multimodal LLMs via Image-to-Text Transformationcited by 124
  32. 2024Efficient Multimodal Learning from Data-centric Perspectivecited by 122
  33. 2024xGen-MM (BLIP-3): A Family of Open Large Multimodal Modelscited by 116
  34. 2024mPLUG-DocOwl2: High-resolution Compressing for OCR-free Multi-page Document Understandingcited by 114
  35. 2023DocPedia: Unleashing the Power of Large Multimodal Model in the Frequency Domain for Versatile Document Understandingcited by 106
  36. 2023Mixture of Cluster-conditional LoRA Experts for Vision-language Instruction Tuningcited by 106
  37. 2024VisualWebBench: How Far Have Multimodal LLMs Evolved in Web Page Understanding and Grounding?cited by 100
  38. 2024Video-CCAM: Enhancing Video-Language Understanding with Causal Cross-Attention Masks for Short and Long Videoscited by 98

50 most cited of 547 All 547 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Model Composition for Multimodal Large Language Models
  2. 2023Macaw-LLM: Multi-Modal Language Modeling with Image, Audio, Video, and Text Integration
  3. 2026Strong Helps Weak: Directional Cross-Modal Alignment Transfer in Multi-modal LLMs
  4. 2025Keeping Yourself is Important in Downstream Tuning Multimodal Large Language Model
  5. 2025SMAR: Soft Modality-Aware Routing Strategy for MoE-based Multimodal Large Language Models Preserving Language Capabilities
  6. 2026MoST: Mixing Speech and Text with Modality-Aware Mixture of Experts
  7. 2023ChatBridge: Bridging Modalities with Large Language Model as a Language Catalyst
  8. 2025FOLDER: Accelerating Multi-modal Large Language Models with Enhanced Performance