mlsanityData to 7 Oct 2026

Training Vision-Language Transformers from Captions

arXiv 2205.09256 ↗First posted 19 May 2022cs.CV cs.MMcited by 11

Not on the map. Why?

Abstract (arXiv)

Vision-Language Transformers can be learned without low-level human labels (e.g. class labels, bounding boxes, etc). Existing work, whether explicitly utilizing bounding boxes or patches, assumes that the visual backbone must first be trained on ImageNet class prediction before being integrated into a multimodal linguistic pipeline. We show that this is not necessary and introduce a new model Vision-Language from Captions (VLC) built on top of Masked Auto-Encoders that does not require this supervision. In fact, in a head-to-head comparison between ViLT, the current state-of-the-art patch-based vision-language transformer which is pretrained with supervised object classification, and our model, VLC, we find that our approach 1. outperforms ViLT on standard benchmarks, 2. provides more interpretable and intuitive patch visualizations, and 3. is competitive with many larger models that utilize ROIs trained on annotated bounding-boxes.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Builds on 65 of its 87 references that are in the map. First posted on arXiv 19 May 2022.

Builds on

65 of 87 references.

References · 65

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  7. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  8. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  9. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  10. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  11. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  12. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
38 more
  1. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  2. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  3. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  4. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  5. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  6. 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,540
  7. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  8. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  9. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,261
  10. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  11. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  12. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,610
  13. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  14. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,471
  15. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  16. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  17. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
  18. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  19. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  20. 2021SimMIM: A Simple Framework for Masked Image Modelingcited by 1,147
  21. 2016Modeling Context in Referring Expressionscited by 1,054
  22. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 966
  23. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  24. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 897
  25. 2021MDETR -- Modulated Detection for End-to-End Multi-Modal Understandingcited by 811
  26. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 645
  27. 2021Masked Feature Prediction for Self-Supervised Visual Pre-Trainingcited by 573
  28. 2021ClipCap: CLIP Prefix for Image Captioningcited by 557
  29. 2022Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionalitycited by 523
  30. 2018A Corpus for Reasoning About Natural Language Grounded in Photographscited by 457
  31. 2018nocaps: novel object captioning at scalecited by 444
  32. 2021SLIP: Self-supervision meets Language-Image Pre-trainingcited by 434
  33. 2021Unifying Vision-and-Language Tasks via Text Generationcited by 411
  34. 2021High-Performance Large-Scale Image Recognition Without Normalizationcited by 359
  35. 2020VirTex: Learning Visual Representations from Textual Annotationscited by 338
  36. 2020Large-Scale Adversarial Training for Vision-and-Language Representation Learningcited by 327
  37. 2020Pixel-BERT: Aligning Image Pixels with Text by Deep Multi-Modal Transformerscited by 288
  38. 2021An Empirical Study of Training End-to-End Vision-and-Language Transformerscited by 271

50 most cited of 65 All 65 (JSON)

Built on by

Cited by year

Citing papers per year: 2022: 5; 2023: 3; 2024: 3.

11 papers.

  1. 2023Large-scale Multi-Modal Pre-trained Models: A Comprehensive Surveycited by 153
  2. 2022Linearly Mapping from Image to Text Spacecited by 139
  3. 2023EVE: Efficient Vision-Language Pre-training with Masked Prediction and Modality-Aware MoEcited by 24
  4. 2024A Survey on Compositional Learning of AI Models: Theoretical and Experimental Practicescited by 15
  5. 2022VLMAE: Vision-Language Masked Autoencodercited by 11
  6. 2022MAMO: Masked Multimodal Modeling for Fine-Grained Vision-Language Representation Learningcited by 11
  7. 2024ECG-Byte: A Tokenizer for End-to-End Generative Electrocardiogram Language Modelingcited by 9
  8. 2024Skews in the Phenomenon Space Hinder Generalization in Text-to-Image Generationcited by 8
  9. 2023Aerial Image Object Detection With Vision Transformer Detector (ViTDet)cited by 3
  10. 2022On Advances in Text Generation from Images Beyond Captioning: A Case Study in Self-Rationalizationcited by 2
  11. 2022EBMs vs. CL: Exploring Self-Supervised Visual Pretraining for Visual Question Answeringcited by 0