mlsanityData to 7 Oct 2026

Pre-training image-language transformers for open-vocabulary tasks

arXiv 2209.04372 ↗First posted 9 Sep 2022cs.CVcited by 11

Not on the map. Why?

Abstract (arXiv)

We present a pre-training approach for vision and language transformer models, which is based on a mixture of diverse tasks. We explore both the use of image-text captioning data in pre-training, which does not need additional supervision, as well as object-aware strategies to pre-train the model. We evaluate the method on a number of textgenerative vision+language tasks, such as Visual Question Answering, visual entailment and captioning, and demonstrate large gains over standard pre-training methods.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Builds on 16 of its 19 references that are in the map. First posted on arXiv 9 Sep 2022.

Builds on

16 of 19 references.

References · 16

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  5. 2015VQA: Visual Question Answeringcited by 2,489
  6. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  7. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  8. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  9. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  10. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 645
  11. 2018The Open Images Dataset V4: Unified image classification, object detection, and visual relationship detection at scalecited by 542
  12. 2021Early Convolutions Help Transformers See Bettercited by 395
4 more
  1. 2019Visual Entailment: A Novel Task for Fine-Grained Image Understandingcited by 250
  2. 2019Connecting Vision and Language with Localized Narrativescited by 225
  3. 2019GQA: A New Dataset for Real-World Visual Reasoning and Compositional Question Answeringcited by 77
  4. 2021Object-aware Video-language Pre-training for Retrievalcited by 57

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 6; 2024: 1; 2025: 2; 2026: 1.

11 papers.

  1. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  2. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 762
  3. 2023PaLI-X: On Scaling up a Multilingual Vision and Language Modelcited by 254
  4. 2024PaliGemma 2: A Family of Versatile VLMs for Transfercited by 204
  5. 2023What Makes for Good Visual Tokenizers for Large Language Models?cited by 42
  6. 2023A Study of Autoregressive Decoders for Multi-Tasking in Computer Visioncited by 10
  7. 2023Leveraging VLM-Based Pipelines to Annotate 3D Objectscited by 10
  8. 2025Adapting Vision-Language Models for Evaluating World Modelscited by 1
  9. 2026Language-Instructed Vision Embeddings for Controllable and Generalizable Perceptioncited by 1
  10. 2023Joint Adaptive Representations for Image-Language Learningcited by 0
  11. 2025Adapting Lightweight Vision Language Models for Radiological Visual Question Answeringcited by 0