mlsanityData to 7 Oct 2026

WenLan: Bridging Vision and Language by Large-Scale Multi-Modal Pre-Training

arXiv 2103.06561 ↗First posted 11 Mar 2021cs.CV cs.IRcited by 84

Not on the map. Why?

Abstract (arXiv)

Multi-modal pre-training models have been intensively explored to bridge vision and language in recent years. However, most of them explicitly model the cross-modal interaction between image-text pairs, by assuming that there exists strong semantic correlation between the text and image modalities. Since this strong assumption is often invalid in real-world scenarios, we choose to implicitly model the cross-modal correlation for large-scale multi-modal pre-training, which is the focus of the Chinese project `WenLan' led by our team. Specifically, with the weak correlation assumption over image-text pairs, we propose a two-tower pre-training model called BriVL within the cross-modal contrastive learning framework. Unlike OpenAI CLIP that adopts a simple contrastive learning method, we devise a more advanced algorithm by adapting the latest method MoCo into the cross-modal scenario. By building a large queue-based dictionary, our BriVL can incorporate more negative samples in limited GPU resources. We further construct a large Chinese multi-source image-text dataset called RUC-CAS-WenLan for pre-training our BriVL model. Extensive experiments demonstrate that the pre-trained BriVL model outperforms both UNITER and OpenAI CLIP on various downstream tasks.

Key facts. Cited by 84 papers in the map (data through 7 Oct 2026). Builds on 36 of its 42 references that are in the map. First posted on arXiv 11 Mar 2021.

Builds on

36 of 42 references.

References · 36

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  6. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  7. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  8. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  9. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  10. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  11. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  12. 2015Fast R-CNNcited by 2,930
24 more
  1. 2020Exploring Simple Siamese Representation Learningcited by 2,853
  2. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  3. 2018Deep contextualized word representationscited by 2,285
  4. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  5. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,838
  6. 2019Self-training with Noisy Student improves ImageNet classificationcited by 1,304
  7. 2018Learning deep representations by mutual information estimation and maximizationcited by 1,265
  8. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  9. 2016Generative Adversarial Text to Image Synthesiscited by 735
  10. 2019Learning Representations by Maximizing Mutual Information Across Viewscited by 733
  11. 2020How Much Knowledge Can You Pack Into the Parameters of a Language Model?cited by 713
  12. 2020VirTex: Learning Visual Representations from Textual Annotationscited by 338
  13. 2020Large-Scale Adversarial Training for Vision-and-Language Representation Learningcited by 327
  14. 2020ERNIE-ViL: Knowledge Enhanced Vision-Language Representations Through Scene Graphcited by 211
  15. 2019Local Aggregation for Unsupervised Learning of Visual Embeddingscited by 204
  16. 2015Semi-supervised Sequence Learningcited by 198
  17. 2020Contrastive learning, multi-view redundancy, and linear modelscited by 146
  18. 2020ImageBERT: Cross-modal Pre-training with Large-scale Weak-supervised Image-Text Datacited by 136
  19. 2017AI Challenger : A Large-scale Dataset for Going Deeper in Image Understandingcited by 73
  20. 2020X-LXMERT: Paint, Caption and Answer Questions with Multi-Modal Transformerscited by 58
  21. 2020XGPT: Cross-modal Generative Pre-Training for Image Captioningcited by 51
  22. 2020Fine-grained Visual Textual Alignment for Cross-Modal Retrieval using Transformer Encoderscited by 47
  23. 2018Recursive Visual Attention in Visual Dialogcited by 34
  24. 2020Multimodal Research in Vision and Language: A Review of Current and Emerging Trendscited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 12; 2022: 32; 2023: 23; 2024: 11; 2025: 5; 2026: 1.

84 papers.

Sort:

By area:

  1. 2021Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigmcited by 405
  2. 2022Multimodal Learning with Transformers: A Surveycited by 399
  3. 2022Prompt-aligned Gradient for Prompt Tuningcited by 339
  4. 2021Pre-Trained Models: Past, Present and Futurecited by 319
  5. 2022X-CLIP: End-to-End Multi-grained Contrastive Learning for Video-Text Retrievalcited by 279
  6. 2023A Comprehensive Survey of AI-Generated Content (AIGC): A History of Generative AI from GAN to ChatGPTcited by 221
  7. 2021Scaling Up Vision-Language Pre-training for Image Captioningcited by 196
  8. 2022A Survey of Vision-Language Pre-Trained Modelscited by 177
  9. 2023Large-scale Multi-Modal Pre-trained Models: A Comprehensive Surveycited by 153
  10. 2024Mobile Edge Intelligence for Large Language Models: A Contemporary Surveycited by 125
  11. 2022Vision-Language Pre-training: Basics, Recent Advances, and Future Trendscited by 117
  12. 2022Wukong: A 100 Million Large-scale Chinese Cross-modal Pre-training Benchmarkcited by 114
38 more
  1. 2023Foundational Models Defining a New Era in Vision: A Survey and Outlookcited by 109
  2. 2023Vision-Language Models in Remote Sensing: Current Progress and Future Trendscited by 93
  3. 2022CenterCLIP: Token Clustering for Efficient Text-Video Retrievalcited by 86
  4. 2022AltCLIP: Altering the Language Encoder in CLIP for Extended Language Capabilitiescited by 84
  5. 2022Open-world Semantic Segmentation via Contrasting and Clustering Vision-Language Embeddingcited by 67
  6. 2022Don't Stop Learning: Towards Continual Learning for the CLIP Modelcited by 61
  7. 2022Long-Form Video-Language Pre-Training with Multimodal Temporal Contrastive Learningcited by 59
  8. 2023FedDAT: An Approach for Foundation Model Finetuning in Multi-Modal Heterogeneous Federated Learningcited by 59
  9. 2023DePT: Decoupled Prompt Tuningcited by 58
  10. 2022Image-text Retrieval: A Survey on Recent Research and Developmentcited by 55
  11. 2021Text-Based Person Search with Limited Datacited by 53
  12. 2021ERNIE-ViLG: Unified Generative Pre-training for Bidirectional Vision-Language Generationcited by 41
  13. 2022Image Difference Captioning with Pre-training and Contrastive Learningcited by 39
  14. 2023Structure-CLIP: Towards Scene Graph Knowledge to Enhance Multi-modal Structured Representationscited by 37
  15. 2021INTERN: A New Learning Paradigm Towards General Visioncited by 33
  16. 2022COTS: Collaborative Two-Stream Vision-Language Pre-Training Model for Cross-Modal Retrievalcited by 33
  17. 2022ViSTA: Vision and Scene Text Aggregation for Cross-Modal Retrievalcited by 29
  18. 2023CTP: Towards Vision-Language Continual Pretraining via Compatible Momentum Contrast and Topology Preservationcited by 29
  19. 2024Few-shot Adaptation of Multi-modal Foundation Models: A Surveycited by 28
  20. 2021Understanding Chinese Video and Language via Contrastive Multimodal Pre-Trainingcited by 23
  21. 2023Cross-modal Contrastive Learning for Multimodal Fake News Detectioncited by 23
  22. 2023Parameter-efficient Tuning of Large-scale Multimodal Foundation Modelcited by 23
  23. 2022COCOA: Cross Modality Contrastive Learning for Sensor Datacited by 21
  24. 2023Synthesizing Long-Term Human Motions with Diffusion Models via Coherent Samplingcited by 19
  25. 2021Curriculum Learning for Vision-and-Language Navigationcited by 18
  26. 2021CMA-CLIP: Cross-Modality Attention CLIP for Image-Text Classificationcited by 16
  27. 2022CODER: Coupled Diversity-Sensitive Momentum Contrastive Learning for Image-Text Retrievalcited by 13
  28. 2023Exploring Large Language Models for Multi-Modal Out-of-Distribution Detectioncited by 12
  29. 2022LGDN: Language-Guided Denoising Network for Video-Language Modelingcited by 11
  30. 2022A survey on knowledge-enhanced multimodal learningcited by 11
  31. 2021EfficientCLIP: Efficient Cross-Modal Pre-training by Ensemble Confident Learning and Language Modelingcited by 10
  32. 2022X2-VLM: All-In-One Pre-trained Model For Vision-Language Taskscited by 9
  33. 2021Multi-modal Self-supervised Pre-training for Regulatory Genome Across Cell Typescited by 8
  34. 2022ERNIE-ViL 2.0: Multi-view Contrastive Learning for Image-Text Pre-trainingcited by 8
  35. 2023Long-Tailed Question Answering in an Open Worldcited by 7
  36. 2025A Closer Look at Conditional Prompt Tuning for Vision-Language Modelscited by 7
  37. 2022Learning by Hallucinating: Vision-Language Pre-training with Weak Supervisioncited by 6
  38. 2022CL-CrossVQA: A Continual Learning Benchmark for Cross-Domain Visual Question Answeringcited by 6

50 most cited of 84 All 84 (JSON)