mlsanityData to 7 Oct 2026

UNIMO-2: End-to-End Unified Vision-Language Grounded Learning

arXiv 2203.09067 ↗First posted 17 Mar 2022cs.CV cs.CLcited by 19

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Vision-Language Pre-training (VLP) has achieved impressive performance on various cross-modal downstream tasks. However, most existing methods can only learn from aligned image-caption data and rely heavily on expensive regional features, which greatly limits their scalability and performance. In this paper, we propose an end-to-end unified-modal pre-training framework, namely UNIMO-2, for joint learning on both aligned image-caption data and unaligned image-only and text-only corpus. We build a unified Transformer model to jointly learn visual representations, textual representations and semantic alignment between images and texts. In particular, we propose to conduct grounded learning on both images and texts via a sharing grounded space, which helps bridge unaligned images and texts, and align the visual and textual semantic spaces on different types of corpora. The experiments show that our grounded learning method can improve textual and visual semantic alignment for improving performance on various cross-modal tasks. Moreover, benefiting from effective joint modeling of different types of corpora, our model also achieves impressive performance on single-modal visual and textual tasks. Our code and models are public at the UNIMO project page https://unimo-ptm.github.io/.

Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 37 of its 56 references that are in the map. First posted on arXiv 17 Mar 2022.

Builds on

37 of 56 references.

Closest ancestors · 3

Same direction, newest first

  1. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervision24 Aug
  2. 2021E2E-VLP: End-to-End Vision-Language Pre-training Enhanced by Visual Learning3 Jun
  3. 2021Seeing Out of tHe bOx: End-to-End Pre-training for Vision-Language Representation Learning7 Apr

Other references · 34

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  6. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  7. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  8. 2017Neural Discrete Representation Learningcited by 4,650
  9. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  10. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  11. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  12. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
22 more
  1. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  2. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  3. 2020Big Self-Supervised Models are Strong Semi-Supervised Learnerscited by 1,375
  4. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,310
  5. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  6. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  7. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  8. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 998
  9. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 966
  10. 2019SpanBERT: Improving Pre-training by Representing and Predicting Spanscited by 832
  11. 2019Unified Language Model Pre-training for Natural Language Understanding and Generationcited by 668
  12. 2018The Open Images Dataset V4: Unified image classification, object detection, and visual relationship detection at scalecited by 542
  13. 2019Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-trainingcited by 451
  14. 2020Large-Scale Adversarial Training for Vision-and-Language Representation Learningcited by 327
  15. 2019Visual Entailment: A Novel Task for Fine-Grained Image Understandingcited by 250
  16. 2020UNIMO: Towards Unified-Modal Understanding and Generation via Cross-Modal Contrastive Learningcited by 239
  17. 2020ERNIE-ViL: Knowledge Enhanced Vision-Language Representations Through Scene Graphcited by 211
  18. 2020Vokenization: Improving Language Understanding with Contextualized, Visual-Grounded Supervisioncited by 93
  19. 2020InterBERT: Vision-and-Language Interaction for Multi-modal Pretrainingcited by 48
  20. 2021Cross-Modal Discrete Representation Learningcited by 35
  21. 2017Learning Visually Grounded Sentence Representationscited by 24
  22. 2015Learning language through picturescited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 4; 2023: 8; 2024: 2; 2025: 3; 2026: 2.

19 papers.

Sort:

By area:

  1. 2022CREPE: Can Vision-Language Foundation Models Reason Compositionally?cited by 177
  2. 2022BridgeTower: Building Bridges Between Encoders in Vision-Language Representation Learningcited by 56
  3. 2023Online Clustered Codebookcited by 55
  4. 2023Revisiting Multimodal Representation in Contrastive Learning: From Patch and Token Embeddings to Finite Discrete Tokenscited by 35
  5. 2022UPainting: Unified Text-to-Image Diffusion Generation with Cross-modal Guidancecited by 30
  6. 2022Single-Stream Multi-Level Alignment for Vision-Language Pretrainingcited by 16
  7. 2024UNIMO-G: Unified Image Generation through Multimodal Conditional Diffusioncited by 15
  8. 2025Bridged Semantic Alignment for Zero-shot 3D Medical Image Diagnosiscited by 9
  9. 2025MGVQ: Could VQ-VAE Beat VAE? A Generalizable Tokenizer with Multi-group Quantizationcited by 9
  10. 2023Masked Autoencoding Does Not Help Natural Language Supervision at Scalecited by 7
  11. 2024Enhancing Fine-Grained Vision-Language Pretraining with Negative Augmented Samplescited by 7
  12. 2023Multimodal Representation Learning of Cardiovascular Magnetic Resonance Imagingcited by 4
7 more
  1. 2023A Survey on Image-text Multimodal Modelscited by 4
  2. 2023PiTL: Cross-modal Retrieval with Weakly-supervised Vision-language Pre-training via Promptingcited by 3
  3. 2023ManagerTower: Aggregating the Insights of Uni-Modal Experts for Vision-Language Representation Learningcited by 2
  4. 2023UNIMO-3: Multi-granularity Interaction for Vision-Language Representation Learningcited by 1
  5. 2026Decoupled Hierarchical Distillation for Multimodal Emotion Recognitioncited by 1
  6. 2025Manager: Aggregating Insights from Unimodal Experts in Two-Tower VLMs and MLLMscited by 0
  7. 2026CodeBind: Decoupled Representation Learning for Multimodal Alignment with Unified Compositional Codebookcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2021SemVLP: Vision-Language Pre-training by Aligning Semantics at Multiple Levels
  2. 2023Stop Pre-Training: Adapt Visual-Language Models to Unseen Languages
  3. 2022VLMixer: Unpaired Vision-Language Pre-training via Cross-Modal CutMix
  4. 2022X2-VLM: All-In-One Pre-trained Model For Vision-Language Tasks
  5. 2024Harnessing Frozen Unimodal Encoders for Flexible Multimodal Alignment
  6. 2022MVPTR: Multi-Level Semantic Alignment for Vision-Language Pre-Training via Multi-Stage Learning
  7. 2021Probing Inter-modality: Visual Parsing with Self-Attention for Vision-Language Pre-training
  8. 2024Assessing and Learning Alignment of Unimodal Vision and Language Models