mlsanityData to 7 Oct 2026

mPLUG-2: A Modularized Multi-modal Foundation Model Across Text, Image and Video

arXiv 2302.00402 ↗First posted 1 Feb 2023cs.CV cs.CL cs.MMcited by 154

Not on the map. Why?

Cited by 43 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Recent years have witnessed a big convergence of language, vision, and multi-modal pretraining. In this work, we present mPLUG-2, a new unified paradigm with modularized design for multi-modal pretraining, which can benefit from modality collaboration while addressing the problem of modality entanglement. In contrast to predominant paradigms of solely relying on sequence-to-sequence generation or encoder-based instance discrimination, mPLUG-2 introduces a multi-module composition network by sharing common universal modules for modality collaboration and disentangling different modality modules to deal with modality entanglement. It is flexible to select different modules for different understanding and generation tasks across all modalities including text, image, and video. Empirical study shows that mPLUG-2 achieves state-of-the-art or competitive results on a broad range of over 30 downstream tasks, spanning multi-modal tasks of image-text and video-text understanding and generation, and uni-modal tasks of text-only, image-only, and video-only understanding. Notably, mPLUG-2 shows new state-of-the-art results of 48.0 top-1 accuracy and 80.3 CIDEr on the challenging MSRVTT video QA and video caption tasks with a far smaller model size and data scale. It also demonstrates strong zero-shot transferability on vision-language and video-language tasks. Code and models will be released in https://github.com/alibaba/AliceMind.

Key facts. Cited by 154 papers in the map (data through 7 Oct 2026). Builds on 104 of its 127 references that are in the map. First posted on arXiv 1 Feb 2023.

Builds on

104 of 127 references.

References · 104

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  7. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  8. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  9. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  10. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  11. 2020End-to-End Object Detection with Transformerscited by 7,384
  12. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
38 more
  1. 2017Mask R-CNNcited by 7,259
  2. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  3. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  4. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  5. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  6. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  7. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  8. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  9. 2022A ConvNet for the 2020scited by 3,545
  10. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  11. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  12. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  13. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
  14. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  15. 2021ViViT: A Video Vision Transformercited by 1,610
  16. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,609
  17. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  18. 2016Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learningcited by 1,509
  19. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  20. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  21. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
  22. 2021Swin Transformer V2: Scaling Up Capacity and Resolutioncited by 1,215
  23. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  24. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  25. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,175
  26. 2021Grounded Language-Image Pre-trainingcited by 1,147
  27. 2019PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarizationcited by 1,100
  28. 2021Video Swin Transformercited by 1,090
  29. 2016Modeling Context in Referring Expressionscited by 1,054
  30. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,041
  31. 2021Scaling Vision Transformerscited by 993
  32. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 966
  33. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  34. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 897
  35. 2021MDETR -- Modulated Detection for End-to-End Multi-Modal Understandingcited by 811
  36. 2021Florence: A New Foundation Model for Computer Visioncited by 802
  37. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
  38. 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 685

50 most cited of 104 All 104 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 49; 2024: 53; 2025: 42; 2026: 10.

154 papers.

Sort:

By area:

  1. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,461
  2. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  3. 2023NExT-GPT: Any-to-Any Multimodal LLMcited by 610
  4. 2023mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaborationcited by 545
  5. 2023InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generationcited by 488
  6. 2024Sora: A Review on Background, Technology, Limitations, and Opportunities of Large Vision Modelscited by 467
  7. 2024Panda-70M: Captioning 70M Videos with Multiple Cross-Modality Teacherscited by 389
  8. 2024InternVideo2: Scaling Foundation Models for Multimodal Video Understandingcited by 310
  9. 2024MA-LMM: Memory-Augmented Large Multimodal Model for Long-Term Video Understandingcited by 223
  10. 2023Any-to-Any Generation via Composable Diffusioncited by 203
  11. 2023Video Understanding with Large Language Models: A Surveycited by 203
  12. 2023Self-Chained Image-Language Model for Video Localization and Question Answeringcited by 200
38 more
  1. 2023Multimodal Large Language Models: A Surveycited by 193
  2. 2023VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and Datasetcited by 168
  3. 2023mPLUG-DocOwl: Modularized Multimodal Large Language Model for Document Understandingcited by 167
  4. 2023Swap Attention in Spatiotemporal Diffusions for Text-to-Video Generationcited by 153
  5. 2024Tarsier: Recipes for Training and Evaluating Large Video Description Modelscited by 140
  6. 2024AuroraCap: Efficient, Performant Video Detailed Captioning and a New Benchmarkcited by 137
  7. 2023PaLI-3 Vision Language Models: Smaller, Faster, Strongercited by 125
  8. 2024Fit and Prune: Fast and Training-free Visual Token Pruning for Multi-modal Large Language Modelscited by 125
  9. 2025Dispider: Enabling Video LLMs with Active Real-Time Interaction via Disentangled Perception, Decision, and Reactioncited by 122
  10. 2023ONE-PEACE: Exploring One General Representation Model Toward Unlimited Modalitiescited by 117
  11. 2024Is Sora a World Simulator? A Comprehensive Survey on General World Models and Beyondcited by 111
  12. 2023Large Models for Time Series and Spatio-Temporal Data: A Survey and Outlookcited by 109
  13. 2024VideoPrism: A Foundational Visual Encoder for Video Understandingcited by 103
  14. 2024Efficient Multimodal Large Language Models: A Surveycited by 95
  15. 2023LION : Empowering Multimodal Large Language Model with Dual-Level Visual Knowledgecited by 80
  16. 2023X-InstructBLIP: A Framework for aligning X-Modal instruction-aware representations to LLMs and Emergent Cross-modal Reasoningcited by 78
  17. 2024MovieChat+: Question-aware Sparse Memory for Long Video Question Answeringcited by 73
  18. 2023RMT: Retentive Networks Meet Vision Transformerscited by 68
  19. 2024Exploring the Frontier of Vision-Language Models: A Survey of Current Methodologies and Future Directionscited by 66
  20. 2023ChatVideo: A Tracklet-centric Multimodal and Versatile Video Understanding Systemcited by 65
  21. 2024I2EBench: A Comprehensive Benchmark for Instruction-based Image Editingcited by 61
  22. 2025DeepVideo-R1: Video Reinforcement Fine-Tuning via Difficulty-aware Regressive GRPOcited by 57
  23. 2024Gramian Multimodal Representation Learning and Alignmentcited by 52
  24. 2024Single Image Unlearning: Efficient Machine Unlearning in Multimodal Large Language Modelscited by 44
  25. 2023Video ChatCaptioner: Towards Enriched Spatiotemporal Descriptionscited by 42
  26. 2023VideoXum: Cross-modal Visual and Textural Summarization of Videoscited by 41
  27. 2024Video-Language Understanding: A Survey from Model Architecture, Model Training, and Data Perspectivescited by 37
  28. 2023GPT4Video: A Unified Multimodal Large Language Model for lnstruction-Followed Understanding and Safety-Aware Generationcited by 36
  29. 2024SimVG: A Simple Framework for Visual Grounding with Decoupled Multi-modal Fusioncited by 36
  30. 2023HowToCaption: Prompting LLMs to Transform Video Annotations at Scalecited by 34
  31. 2025MeViS: A Multi-Modal Dataset for Referring Motion Expression Video Segmentationcited by 34
  32. 2023Youku-mPLUG: A 10 Million Large-scale Chinese Video-Language Dataset for Pre-training and Benchmarkscited by 33
  33. 2025MemVerse: Multimodal Memory for Lifelong Learning Agentscited by 31
  34. 2024Review of multimodal machine learning approaches in healthcarecited by 30
  35. 2023VideoCon: Robust Video-Language Alignment via Contrast Captionscited by 29
  36. 2023Omni-SMoLA: Boosting Generalist Multimodal Models with Soft Mixture of Low-rank Expertscited by 26
  37. 2024FINECAPTION: Compositional Image Captioning Focusing on Wherever You Want at Any Granularitycited by 26
  38. 2025IteRPrimE: Zero-shot Referring Image Segmentation with Iterative Grad-CAM Refinement and Primary Word Emphasiscited by 26

50 most cited of 154 All 154 (JSON)