mlsanityData to 14 Oct 2026

UNIMO: Towards Unified-Modal Understanding and Generation via Cross-Modal Contrastive Learning

arXiv 2012.15409 ↗First posted 31 Dec 2020cs.CLcited by 239

Not on the map. Why?

Cited by 68 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Existed pre-training methods either focus on single-modal tasks or multi-modal tasks, and cannot effectively adapt to each other. They can only utilize single-modal data (i.e. text or image) or limited multi-modal data (i.e. image-text pairs). In this work, we propose a unified-modal pre-training architecture, namely UNIMO, which can effectively adapt to both single-modal and multi-modal understanding and generation tasks. Large scale of free text corpus and image collections can be utilized to improve the capability of visual and textual understanding, and cross-modal contrastive learning (CMCL) is leveraged to align the textual and visual information into a unified semantic space over a corpus of image-text pairs. As the non-paired single-modal data is very rich, our model can utilize much larger scale of data to learn more generalizable representations. Moreover, the textual knowledge and visual knowledge can enhance each other in the unified semantic space. The experimental results show that UNIMO significantly improves the performance of several single-modal and multi-modal downstream tasks. Our code and pre-trained models are public at the UNIMO project page https://unimo-ptm.github.io/

Key facts. Cited by 239 papers in the map (data through 14 Oct 2026). First posted on arXiv 31 Dec 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 26; 2022: 96; 2023: 82; 2024: 24; 2025: 6; 2026: 5.

239 papers.

Sort:

By area:

  1. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  2. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,224
  3. 2021Florence: A New Foundation Model for Computer Visioncited by 804
  4. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 745
  5. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 649
  6. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 647
  7. 2021FILIP: Fine-grained Interactive Language-Image Pre-Trainingcited by 627
  8. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 537
  9. 2021VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Expertscited by 425
  10. 2021A Survey of Transformerscited by 401
  11. 2022Multimodal Learning with Transformers: A Surveycited by 400
  12. 2021An Empirical Study of Training End-to-End Vision-and-Language Transformerscited by 271
38 more
  1. 2021Multi-Grained Vision Language Pre-Training: Aligning Texts with Visual Conceptscited by 256
  2. 2022Unified Structure Generation for Universal Information Extractioncited by 221
  3. 2022Vision-Language Pre-Training with Triple Contrastive Learningcited by 212
  4. 2023ULIP-2: Towards Scalable Multimodal Pre-training for 3D Understandingcited by 202
  5. 2021Scaling Up Vision-Language Pre-training for Image Captioningcited by 196
  6. 2022mPLUG: Effective and Efficient Vision-Language Learning by Cross-modal Skip-connectionscited by 187
  7. 2022MaskCLIP: Masked Self-Distillation Advances Contrastive Language-Image Pretrainingcited by 187
  8. 2021CPT: Colorful Prompt Tuning for Pre-trained Vision-Language Modelscited by 183
  9. 2022A Survey of Vision-Language Pre-Trained Modelscited by 177
  10. 2022All in One: Exploring Unified Video-Language Pre-trainingcited by 171
  11. 2022SegCLIP: Patch Aggregation with Learnable Centers for Open-Vocabulary Semantic Segmentationcited by 164
  12. 2023mPLUG-2: A Modularized Multi-modal Foundation Model Across Text, Image and Videocited by 154
  13. 2023Large-scale Multi-Modal Pre-trained Models: A Comprehensive Surveycited by 153
  14. 2022OmniVL:One Foundation Model for Image-Language and Video-Language Taskscited by 143
  15. 2022Revealing Single Frame Bias for Video-and-Language Learningcited by 133
  16. 2023ONE-PEACE: Exploring One General Representation Model Toward Unlimited Modalitiescited by 118
  17. 2022Coarse-to-Fine Vision-Language Pre-training with Fusion in the Backbonecited by 109
  18. 2022PyramidCLIP: Hierarchical Feature Alignment for Vision-language Model Pretrainingcited by 104
  19. 2021A Good Prompt Is Worth Millions of Parameters: Low-resource Prompt-based Learning for Vision-Language Modelscited by 101
  20. 2022Retrieval Augmented Visual Question Answering with Outside Knowledgecited by 94
  21. 2023Multimodality Helps Unimodality: Cross-Modal Few-Shot Learning with Multimodal Modelscited by 91
  22. 2022Plug-and-Play VQA: Zero-shot VQA by Conjoining Large Pretrained Models with Zero Trainingcited by 88
  23. 2023Scaling Vision-Language Models with Sparse Mixture of Expertscited by 88
  24. 2023Tag2Text: Guiding Vision-Language Model via Image Taggingcited by 86
  25. 2024Natural Language Understanding and Inference with MLLM in Visual Question Answering: A Surveycited by 76
  26. 2022Multi-Modal Knowledge Graph Construction and Application: A Surveycited by 74
  27. 2022Plausible May Not Be Faithful: Probing Object Hallucination in Vision-Language Pre-trainingcited by 72
  28. 2022HiTeA: Hierarchical Temporal-Aware Video-Language Pre-trainingcited by 72
  29. 2022Enabling Multimodal Generation on CLIP via Vision-Language Knowledge Distillationcited by 70
  30. 2022Fine-Grained Semantically Aligned Vision-Language Pre-Trainingcited by 68
  31. 2022Open-world Semantic Segmentation via Contrasting and Clustering Vision-Language Embeddingcited by 67
  32. 2022CLIP-TSA: CLIP-Assisted Temporal Self-Attention for Weakly-Supervised Video Anomaly Detectioncited by 67
  33. 2022Multi-modal Alignment using Representation Codebookcited by 63
  34. 2022Cross-modal Contrastive Learning for Speech Translationcited by 61
  35. 2022MixGen: A New Multi-Modal Data Augmentationcited by 61
  36. 2023Zero-shot Referring Image Segmentation with Global-Local Context Featurescited by 61
  37. 2023Self-Supervised Multimodal Learning: A Surveycited by 58
  38. 2024MyVLM: Personalizing VLMs for User-Specific Queriescited by 58

50 most cited of 239 All 239 (JSON)