mlsanityData to 7 Oct 2026

Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks

arXiv 2004.06165 ↗First posted 13 Apr 2020cs.CV cs.CL cs.IRcited by 1,213

Not on the map. Why?

Cited by 266 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Large-scale pre-training methods of learning cross-modal representations on image-text pairs are becoming popular for vision-language tasks. While existing methods simply concatenate image region features and text features as input to the model to be pre-trained and use self-attention to learn image-text semantic alignments in a brute force manner, in this paper, we propose a new learning method Oscar (Object-Semantics Aligned Pre-training), which uses object tags detected in images as anchor points to significantly ease the learning of alignments. Our method is motivated by the observation that the salient objects in an image can be accurately detected, and are often mentioned in the paired text. We pre-train an Oscar model on the public corpus of 6.5 million text-image pairs, and fine-tune it on downstream tasks, creating new state-of-the-arts on six well-established vision-language understanding and generation tasks.

Key facts. Cited by 1,213 papers in the map (data through 7 Oct 2026). First posted on arXiv 13 Apr 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 213; 2022: 364; 2023: 355; 2024: 169; 2025: 87; 2026: 25.

1,213 papers.

Sort:

By area:

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  3. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  4. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  5. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  6. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  7. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  8. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  9. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  10. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  11. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
  12. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
38 more
  1. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,197
  2. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,175
  3. 2021Grounded Language-Image Pre-trainingcited by 1,147
  4. 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapterscited by 1,021
  5. 2021Transformers in Vision: A Surveycited by 1,009
  6. 2021StyleCLIP: Text-Driven Manipulation of StyleGAN Imagerycited by 990
  7. 2021MDETR -- Modulated Detection for End-to-End Multi-Modal Understandingcited by 811
  8. 2021Florence: A New Foundation Model for Computer Visioncited by 802
  9. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
  10. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Modelscited by 659
  11. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 657
  12. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 646
  13. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 645
  14. 2021FILIP: Fine-grained Interactive Language-Image Pre-Trainingcited by 626
  15. 2021RegionCLIP: Region-based Language-Image Pretrainingcited by 612
  16. 2023NExT-GPT: Any-to-Any Multimodal LLMcited by 610
  17. 2021ClipCap: CLIP Prefix for Image Captioningcited by 557
  18. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 536
  19. 2022GroupViT: Semantic Segmentation Emerges from Text Supervisioncited by 529
  20. 2022Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Taskscited by 510
  21. 2024MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmarkcited by 494
  22. 2021Less is More: ClipBERT for Video-and-Language Learning via Sparse Samplingcited by 484
  23. 2021LeViT: a Vision Transformer in ConvNet's Clothing for Faster Inferencecited by 475
  24. 2021VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Expertscited by 424
  25. 2021Unifying Vision-and-Language Tasks via Text Generationcited by 411
  26. 2021Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigmcited by 405
  27. 2022Multimodal Learning with Transformers: A Surveycited by 399
  28. 2021ScanQA: 3D Question Answering for Spatial Scene Understandingcited by 340
  29. 2021How Much Can CLIP Benefit Vision-and-Language Tasks?cited by 321
  30. 2021Pre-Trained Models: Past, Present and Futurecited by 319
  31. 2021Generic Attention-model Explainability for Interpreting Bi-Modal and Encoder-Decoder Transformerscited by 316
  32. 2022SQA3D: Situated Question Answering in 3D Scenescited by 303
  33. 2021An Empirical Study of GPT-3 for Few-Shot Knowledge-Based VQAcited by 298
  34. 2023Side Adapter Network for Open-Vocabulary Semantic Segmentationcited by 296
  35. 2022Generalized Decoding for Pixel, Image, and Languagecited by 292
  36. 2022X-CLIP: End-to-End Multi-grained Contrastive Learning for Video-Text Retrievalcited by 279
  37. 2024MM-LLMs: Recent Advances in MultiModal Large Language Modelscited by 277
  38. 2023EmbodiedGPT: Vision-Language Pre-Training via Embodied Chain of Thoughtcited by 276

50 most cited of 1,213 All 1,213 (JSON)