mlsanityData to 7 Oct 2026

mPLUG: Effective and Efficient Vision-Language Learning by Cross-modal Skip-connections

arXiv 2205.12005 ↗First posted 24 May 2022cs.CL cs.CVcited by 185

Not on the map. Why?

Cited by 45 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Large-scale pretrained foundation models have been an emerging paradigm for building artificial intelligence (AI) systems, which can be quickly adapted to a wide range of downstream tasks. This paper presents mPLUG, a new vision-language foundation model for both cross-modal understanding and generation. Most existing pre-trained models suffer from the problems of low computational efficiency and information asymmetry brought by the long visual sequence in cross-modal alignment. To address these problems, mPLUG introduces an effective and efficient vision-language architecture with novel cross-modal skip-connections, which creates inter-layer shortcuts that skip a certain number of layers for time-consuming full self-attention on the vision side. mPLUG is pre-trained end-to-end on large-scale image-text pairs with both discriminative and generative objectives. It achieves state-of-the-art results on a wide range of vision-language downstream tasks, such as image captioning, image-text retrieval, visual grounding and visual question answering. mPLUG also demonstrates strong zero-shot transferability when directly transferred to multiple video-language tasks.

Key facts. Cited by 185 papers in the map (data through 7 Oct 2026). Builds on 65 of its 85 references that are in the map. First posted on arXiv 24 May 2022.

Builds on

65 of 85 references.

References · 65

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  7. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  8. 2016Densely Connected Convolutional Networkscited by 6,691
  9. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  10. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  11. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  12. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
38 more
  1. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  2. 2015VQA: Visual Question Answeringcited by 2,489
  3. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,261
  4. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  5. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  6. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  7. 2016Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learningcited by 1,509
  8. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  9. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  10. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,310
  11. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  12. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  13. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  14. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  15. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,175
  16. 2016Modeling Context in Referring Expressionscited by 1,054
  17. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 966
  18. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  19. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 946
  20. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 897
  21. 2016Training Deep Nets with Sublinear Memory Costcited by 825
  22. 2021MDETR -- Modulated Detection for End-to-End Multi-Modal Understandingcited by 811
  23. 2021Florence: A New Foundation Model for Computer Visioncited by 802
  24. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 645
  25. 2021FILIP: Fine-grained Interactive Language-Image Pre-Trainingcited by 626
  26. 2021VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understandingcited by 596
  27. 2016Self-critical Sequence Training for Image Captioningcited by 544
  28. 2019End-to-End Learning of Visual Representations from Uncurated Instructional Videoscited by 506
  29. 2018A Corpus for Reasoning About Natural Language Grounded in Photographscited by 457
  30. 2018nocaps: novel object captioning at scalecited by 444
  31. 2021VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Textcited by 427
  32. 2021VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Expertscited by 424
  33. 2021Unifying Vision-and-Language Tasks via Text Generationcited by 411
  34. 2020Large-Scale Adversarial Training for Vision-and-Language Representation Learningcited by 327
  35. 2021How Much Can CLIP Benefit Vision-and-Language Tasks?cited by 321
  36. 2021MERLOT: Multimodal Neural Script Knowledge Modelscited by 311
  37. 2020Pixel-BERT: Aligning Image Pixels with Text by Deep Multi-Modal Transformerscited by 288
  38. 2021An Empirical Study of Training End-to-End Vision-and-Language Transformerscited by 271

50 most cited of 65 All 65 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 14; 2023: 63; 2024: 59; 2025: 42; 2026: 7.

185 papers.

Sort:

By area:

  1. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,461
  2. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  3. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 560
  4. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 536
  5. 2024ELLA: Equip Diffusion Models with LLM for Enhanced Semantic Alignmentcited by 455
  6. 2022Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Taskscited by 420
  7. 2023TIFA: Accurate and Interpretable Text-to-Image Faithfulness Evaluation with Question Answeringcited by 416
  8. 2023mPLUG-DocOwl: Modularized Multimodal Large Language Model for Document Understandingcited by 167
  9. 2023Davidsonian Scene Graph: Improving Reliability in Fine-grained Evaluation for Text-to-Image Generationcited by 159
  10. 2023mPLUG-2: A Modularized Multi-modal Foundation Model Across Text, Image and Videocited by 154
  11. 2023On the Adversarial Robustness of Multi-Modal Foundation Modelscited by 142
  12. 2025RoboBrain: A Unified Brain Model for Robotic Manipulation from Abstract to Concretecited by 134
38 more
  1. 2022Vision-Language Pre-training: Basics, Recent Advances, and Future Trendscited by 117
  2. 2022Chinese CLIP: Contrastive Vision-Language Pretraining in Chinesecited by 108
  3. 2023Navigating Text-To-Image Customization: From LyCORIS Fine-Tuning to Model Evaluationcited by 108
  4. 2025Mogao: An Omni Foundation Model for Interleaved Multi-Modal Generationcited by 108
  5. 2022A Survey of Knowledge Graph Reasoning on Graph Types: Static, Dynamic, and Multimodalcited by 106
  6. 2023StarVector: Generating Scalable Vector Graphics Code from Images and Textcited by 96
  7. 2023Expressive Text-to-Image Generation with Rich Textcited by 89
  8. 2023Divide & Bind Your Attention for Improved Generative Semantic Nursingcited by 80
  9. 2023X-InstructBLIP: A Framework for aligning X-Modal instruction-aware representations to LLMs and Emergent Cross-modal Reasoningcited by 78
  10. 2023PanoGen: Text-Conditioned Panoramic Environment Generation for Vision-and-Language Navigationcited by 77
  11. 2022HiTeA: Hierarchical Temporal-Aware Video-Language Pre-trainingcited by 72
  12. 2024Vript: A Video Is Worth Thousands of Wordscited by 71
  13. 2025TIIF-Bench: How Does Your T2I Model Follow Your Instructions?cited by 64
  14. 2024OpenBias: Open-set Bias Detection in Text-to-Image Generative Modelscited by 61
  15. 2022BridgeTower: Building Bridges Between Encoders in Vision-Language Representation Learningcited by 56
  16. 2024Faster Diffusion via Temporal Attention Decompositioncited by 54
  17. 2024Toward Visual Grounding: A Surveycited by 52
  18. 2024Towards A Better Metric for Text-to-Video Generationcited by 50
  19. 2023FaithScore: Fine-grained Evaluations of Hallucinations in Large Vision-Language Modelscited by 44
  20. 2023If at First You Don't Succeed, Try, Try Again: Faithful Diffusion-based Text-to-Image Generation by Selectioncited by 40
  21. 2024PromptFix: You Prompt and We Fix the Photocited by 40
  22. 2024Core Knowledge Deficits in Multi-Modal Language Modelscited by 38
  23. 2023FunQA: Towards Surprising Video Comprehensioncited by 35
  24. 2023HowToCaption: Prompting LLMs to Transform Video Annotations at Scalecited by 34
  25. 2023Youku-mPLUG: A 10 Million Large-scale Chinese Video-Language Dataset for Pre-training and Benchmarkscited by 33
  26. 2024Scalable Ranked Preference Optimization for Text-to-Image Generationcited by 33
  27. 2024Question Aware Vision Transformer for Multimodal Reasoningcited by 32
  28. 2023RAMM: Retrieval-augmented Biomedical Visual Question Answering with Multi-modal Pre-trainingcited by 31
  29. 2023How to Configure Good In-Context Sequence for Visual Question Answeringcited by 31
  30. 2025Selftok: Discrete Visual Tokens of Autoregression, by Diffusion, and for Reasoningcited by 31
  31. 2023Benchmarking Robustness of Adaptation Methods on Pre-trained Vision-Language Modelscited by 30
  32. 2024Review of multimodal machine learning approaches in healthcarecited by 30
  33. 2024Prior Knowledge Integration via LLM Encoding and Pseudo Event Regulation for Video Moment Retrievalcited by 30
  34. 2024Agent3D-Zero: An Agent for Zero-shot 3D Understandingcited by 29
  35. 2023Bootstrapping Vision-Language Learning with Decoupled Language Pre-trainingcited by 28
  36. 2024Break the Visual Perception: Adversarial Attacks Targeting Encoded Visual Tokens of Large Vision-Language Modelscited by 27
  37. 2024SnapGen: Taming High-Resolution Text-to-Image Models for Mobile Devices with Efficient Architectures and Trainingcited by 27
  38. 2022Named Entity and Relation Extraction with Multi-Modal Retrievalcited by 26

50 most cited of 185 All 185 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024mPLUG-Owl3: Towards Long Image-Sequence Understanding in Multi-Modal Large Language Models
  2. 2023Stop Pre-Training: Adapt Visual-Language Models to Unseen Languages
  3. 2021SemVLP: Vision-Language Pre-training by Aligning Semantics at Multiple Levels
  4. 2023Joint Adaptive Representations for Image-Language Learning
  5. 2022ERNIE-UniX2: A Unified Cross-lingual Cross-modal Framework for Understanding and Generation
  6. 2024CMAL: A Novel Cross-Modal Associative Learning Framework for Vision-Language Pre-Training
  7. 2022Vision-Language Intelligence: Tasks, Representation Learning, and Large Models
  8. 2025Improved Alignment of Modalities in Large Vision Language Models