mlsanityData to 7 Oct 2026

mPLUG-Owl: Modularization Empowers Large Language Models with Multimodality

arXiv 2304.14178 ↗First posted 27 Apr 2023cs.CL cs.CV cs.LGcited by 1,038

Not on the map. Why?

Cited by 617 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Large language models (LLMs) have demonstrated impressive zero-shot abilities on a variety of open-ended tasks, while recent research has also explored the use of LLMs for multi-modal generation. In this study, we introduce mPLUG-Owl, a novel training paradigm that equips LLMs with multi-modal abilities through modularized learning of foundation LLM, a visual knowledge module, and a visual abstractor module. This approach can support multiple modalities and facilitate diverse unimodal and multimodal abilities through modality collaboration. The training paradigm of mPLUG-Owl involves a two-stage method for aligning image and text, which learns visual knowledge with the assistance of LLM while maintaining and even improving the generation abilities of LLM. In the first stage, the visual knowledge module and abstractor module are trained with a frozen LLM module to align the image and text. In the second stage, language-only and multi-modal supervised datasets are used to jointly fine-tune a low-rank adaption (LoRA) module on LLM and the abstractor module by freezing the visual knowledge module. We carefully build a visually-related instruction evaluation set OwlEval. Experimental results show that our model outperforms existing multi-modal models, demonstrating mPLUG-Owl's impressive instruction and visual understanding ability, multi-turn conversation ability, and knowledge reasoning ability. Besides, we observe some unexpected and exciting abilities such as multi-image correlation and scene text understanding, which makes it possible to leverage it for harder real scenarios, such as vision-only document comprehension. Our code, pre-trained model, instruction-tuned models, and evaluation set are available at https://github.com/X-PLUG/mPLUG-Owl. The online demo is available at https://www.modelscope.cn/studios/damo/mPLUG-Owl.

Key facts. Cited by 1,038 papers in the map (data through 7 Oct 2026). Builds on 27 of its 36 references that are in the map. First posted on arXiv 27 Apr 2023.

Builds on

27 of 36 references.

References · 27

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2023GPT-4 Technical Reportcited by 18,132
  5. 2022Training language models to follow instructions with human feedbackcited by 15,712
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  7. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  8. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  9. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  10. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  11. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  12. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
15 more
  1. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  2. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  3. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  4. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  5. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  6. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  7. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  8. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,173
  9. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Modelscited by 659
  10. 2023MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Actioncited by 511
  11. 2023Multimodal C4: An Open, Billion-scale Corpus of Images Interleaved with Textcited by 208
  12. 2022mPLUG: Effective and Efficient Vision-Language Learning by Cross-modal Skip-connectionscited by 185
  13. 2023mPLUG-2: A Modularized Multi-modal Foundation Model Across Text, Image and Videocited by 154
  14. 2021E2E-VLP: End-to-End Vision-Language Pre-training Enhanced by Visual Learningcited by 84
  15. 2022HiTeA: Hierarchical Temporal-Aware Video-Language Pre-trainingcited by 72

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 252; 2024: 473; 2025: 242; 2026: 71.

1,038 papers.

Sort:

By area:

  1. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  2. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  3. 2023Qwen Technical Reportcited by 3,273
  4. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  5. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  6. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  7. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  8. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
  9. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,582
  10. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,461
  11. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,385
  12. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
38 more
  1. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,086
  2. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
  3. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,000
  4. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 929
  5. 2023A Survey on Multimodal Large Language Modelscited by 898
  6. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 893
  7. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 891
  8. 2023VideoChat: Chat-Centric Video Understandingcited by 851
  9. 2023LISA: Reasoning Segmentation via Large Language Modelcited by 770
  10. 2024VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMscited by 749
  11. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 703
  12. 2023Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Modelscited by 695
  13. 2023VILA: On Pre-training for Visual Language Modelscited by 686
  14. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)cited by 677
  15. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decodingcited by 671
  16. 2023EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understandingcited by 666
  17. 2023A Comprehensive Overview of Large Language Modelscited by 639
  18. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 617
  19. 2023NExT-GPT: Any-to-Any Multimodal LLMcited by 610
  20. 2024MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?cited by 573
  21. 2023HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Modelscited by 572
  22. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 560
  23. 2024SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agentscited by 553
  24. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuningcited by 548
  25. 2023mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaborationcited by 545
  26. 2023MovieChat: From Dense Token to Sparse Memory for Long Video Understandingcited by 536
  27. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularitycited by 498
  28. 2024MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmarkcited by 494
  29. 2023OCRBench: On the Hidden Mystery of OCR in Large Multimodal Modelscited by 466
  30. 2023GLaMM: Pixel Grounding Large Multimodal Modelcited by 447
  31. 2023Generative Multimodal Models are In-Context Learnerscited by 446
  32. 2023Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuningcited by 439
  33. 2023RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedbackcited by 404
  34. 2025DeepEyes: Incentivizing "Thinking with Images" via Reinforcement Learningcited by 398
  35. 2024ShareGPT4Video: Improving Video Understanding and Generation with Better Captionscited by 386
  36. 2024Hallucination of Multimodal Large Language Models: A Surveycited by 380
  37. 2023Chat-UniVi: Unified Visual Representation Empowers Large Language Models with Image and Video Understandingcited by 368
  38. 2023LanguageBind: Extending Video-Language Pretraining to N-modality by Language-based Semantic Alignmentcited by 366

50 most cited of 1,038 All 1,038 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023MM-LIMA: Less Is More for Alignment in Multi-Modal Datasets
  2. 2025Efficiently Integrate Large Language Models with Visual Perception: A Survey from the Training Paradigm Perspective
  3. 2023SwitchGPT: Adapting Large Language Models for Non-Text Outputs
  4. 2023InfMLLM: A Unified Framework for Visual-Language Tasks
  5. 2023X-LLM: Bootstrapping Advanced Large Language Models by Treating Multi-Modalities as Foreign Languages
  6. 2024MM1.5: Methods, Analysis & Insights from Multimodal LLM Fine-tuning
  7. 2024Semantic Alignment for Multimodal Large Language Models
  8. 2024LMFusion: Adapting Pretrained Language Models for Multimodal Generation