mlsanityData to 7 Oct 2026

ImageBERT: Cross-modal Pre-training with Large-scale Weak-supervised Image-Text Data

arXiv 2001.07966 ↗First posted 22 Jan 2020cs.CVcited by 136

Not on the map. Why?

Cited by 31 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

In this paper, we introduce a new vision-language pre-trained model -- ImageBERT -- for image-text joint embedding. Our model is a Transformer-based model, which takes different modalities as input and models the relationship between them. The model is pre-trained on four tasks simultaneously: Masked Language Modeling (MLM), Masked Object Classification (MOC), Masked Region Feature Regression (MRFR), and Image Text Matching (ITM). To further enhance the pre-training quality, we have collected a Large-scale weAk-supervised Image-Text (LAIT) dataset from Web. We first pre-train the model on this dataset, then conduct a second stage pre-training on Conceptual Captions and SBU Captions. Our experiments show that multi-stage pre-training strategy outperforms single-stage pre-training. We also fine-tune and evaluate our pre-trained ImageBERT model on image retrieval and text retrieval tasks, and achieve new state-of-the-art results on both MSCOCO and Flickr30k datasets.

Key facts. Cited by 136 papers in the map (data through 7 Oct 2026). First posted on arXiv 22 Jan 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 32; 2022: 39; 2023: 40; 2024: 19; 2025: 5; 2026: 1.

136 papers.

Sort:

By area:

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  3. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  4. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  5. 2021Florence: A New Foundation Model for Computer Visioncited by 802
  6. 2021FILIP: Fine-grained Interactive Language-Image Pre-Trainingcited by 626
  7. 2022TransFuser: Imitation with Transformer-Based Sensor Fusion for Autonomous Drivingcited by 477
  8. 2022Multimodal Learning with Transformers: A Surveycited by 399
  9. 2021Multi-Modal Fusion Transformer for End-to-End Autonomous Drivingcited by 372
  10. 2021Pre-Trained Models: Past, Present and Futurecited by 319
  11. 2021WIT: Wikipedia-based Image Text Dataset for Multimodal Multilingual Machine Learningcited by 285
  12. 2023WavCaps: A ChatGPT-Assisted Weakly-Labelled Audio Captioning Dataset for Audio-Language Multimodal Researchcited by 266
38 more
  1. 2022Vision-Language Pre-Training with Triple Contrastive Learningcited by 212
  2. 2021VisualGPT: Data-efficient Adaptation of Pretrained Language Models for Image Captioningcited by 190
  3. 2022MaskCLIP: Masked Self-Distillation Advances Contrastive Language-Image Pretrainingcited by 187
  4. 2021RedCaps: web-curated image-text data created by the people, for the peoplecited by 175
  5. 2023Towards Open Vocabulary Learning: A Surveycited by 165
  6. 2022VLP: A Survey on Vision-Language Pre-trainingcited by 159
  7. 2023Large-scale Multi-Modal Pre-trained Models: A Comprehensive Surveycited by 153
  8. 2022StyleGAN-Human: A Data-Centric Odyssey of Human Generationcited by 143
  9. 2021Recent Advances in Deep Learning Based Dialogue Systems: A Systematic Surveycited by 130
  10. 2024Mobile Edge Intelligence for Large Language Models: A Contemporary Surveycited by 125
  11. 2021Uni-Perceiver: Pre-training Unified Architecture for Generic Perception for Zero-shot and Few-shot Taskscited by 124
  12. 2021Multi-modal Understanding and Generation for Medical Images and Text via Vision-Language Pre-Trainingcited by 96
  13. 2023TinyCLIP: CLIP Distillation via Affinity Mimicking and Weight Inheritancecited by 90
  14. 2021WenLan: Bridging Vision and Language by Large-Scale Multi-Modal Pre-Trainingcited by 84
  15. 2024Natural Language Understanding and Inference with MLLM in Visual Question Answering: A Surveycited by 74
  16. 2022Fine-Grained Semantically Aligned Vision-Language Pre-Trainingcited by 68
  17. 2021Kaleido-BERT: Vision-Language Pre-training on Fashion Domaincited by 67
  18. 2022Masked Vision and Language Modeling for Multi-modal Representation Learningcited by 66
  19. 2022Multi-modal Alignment using Representation Codebookcited by 63
  20. 2022MixGen: A New Multi-Modal Data Augmentationcited by 61
  21. 2022Uni-Perceiver-MoE: Learning Sparse Generalist Models with Conditional MoEscited by 58
  22. 2023Learning Customized Visual Models with Retrieval-Augmented Knowledgecited by 58
  23. 2022Image-text Retrieval: A Survey on Recent Research and Developmentcited by 55
  24. 2023RenAIssance: A Survey into AI Text-to-Image Generation in the Era of Large Modelcited by 47
  25. 2022Image Difference Captioning with Pre-training and Contrastive Learningcited by 39
  26. 2023VidChapters-7M: Video Chapters at Scalecited by 39
  27. 2023Transformers in Speech Processing: A Surveycited by 38
  28. 2022Show Me What and Tell Me How: Video Synthesis via Multimodal Conditioningcited by 37
  29. 2021M6-UFC: Unifying Multi-Modal Controls for Conditional Image Synthesis via Non-Autoregressive Generative Transformerscited by 36
  30. 2021Data Efficient Language-supervised Zero-shot Recognition with Optimal Transport Distillationcited by 35
  31. 2021Video Transformer for Deepfake Detection with Incremental Learningcited by 33
  32. 2022FashionViL: Fashion-Focused Vision-and-Language Representation Learningcited by 32
  33. 2023GPT-4 Enhanced Multimodal Grounding for Autonomous Driving: Leveraging Cross-Modal Attention with Large Language Modelscited by 31
  34. 2022ViSTA: Vision and Scene Text Aggregation for Cross-Modal Retrievalcited by 29
  35. 2022Dynamic MDETR: A Dynamic Multimodal Transformer Decoder for Visual Groundingcited by 29
  36. 2022End-to-end Dense Video Captioning as Sequence Generationcited by 27
  37. 2022Position-guided Text Prompt for Vision-Language Pre-trainingcited by 27
  38. 2021M5Product: Self-harmonized Contrastive Learning for E-commercial Multi-modal Pretrainingcited by 26

50 most cited of 136 All 136 (JSON)