mlsanityData to 7 Oct 2026

Unifying Vision-Language Representation Space with Single-tower Transformer

arXiv 2211.11153 ↗First posted 21 Nov 2022cs.LG cs.CL cs.CVcited by 21

Not on the map. Why?

Abstract (arXiv)

Contrastive learning is a form of distance learning that aims to learn invariant features from two related representations. In this paper, we explore the bold hypothesis that an image and its caption can be simply regarded as two different views of the underlying mutual information, and train a model to learn a unified vision-language representation space that encodes both modalities at once in a modality-agnostic manner. We first identify difficulties in learning a generic one-tower model for vision-language pretraining (VLP), and propose OneR as a simple yet effective framework for our goal. We discover intriguing properties that distinguish OneR from the previous works that learn modality-specific representation spaces such as zero-shot object localization, text-guided visual reasoning and multi-modal retrieval, and present analyses to provide insights into this new form of multi-modal representation learning. Thorough evaluations demonstrate the potential of a unified modality-agnostic VLP framework.

Key facts. Cited by 21 papers in the map (data through 7 Oct 2026). Builds on 42 of its 48 references that are in the map. First posted on arXiv 21 Nov 2022.

Builds on

42 of 48 references.

References · 42

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  6. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  7. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  8. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  9. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  10. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  11. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,690
  12. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
30 more
  1. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  2. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  3. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  4. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  5. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  6. 2020Exploring Simple Siamese Representation Learningcited by 2,853
  7. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  8. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  9. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,610
  10. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,479
  11. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
  12. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  13. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  14. 2021Florence: A New Foundation Model for Computer Visioncited by 802
  15. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 646
  16. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 645
  17. 2021ClipCap: CLIP Prefix for Image Captioningcited by 557
  18. 2021SLIP: Self-supervision meets Language-Image Pre-trainingcited by 434
  19. 2022Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Taskscited by 420
  20. 2021Early Convolutions Help Transformers See Bettercited by 395
  21. 2020Large-Scale Adversarial Training for Vision-and-Language Representation Learningcited by 327
  22. 2014One weird trick for parallelizing convolutional neural networkscited by 264
  23. 2020UNIMO: Towards Unified-Modal Understanding and Generation via Cross-Modal Contrastive Learningcited by 239
  24. 2019UNITER: UNiversal Image-TExt Representation Learningcited by 225
  25. 2022Vision-Language Pre-Training with Triple Contrastive Learningcited by 212
  26. 2022Multimodal Contrastive Learning with LIMoE: the Language-Image Mixture of Expertscited by 202
  27. 2021Uni-Perceiver: Pre-training Unified Architecture for Generic Perception for Zero-shot and Few-shot Taskscited by 124
  28. 2022MixGen: A New Multi-Modal Data Augmentationcited by 61
  29. 2021UFO: A UniFied TransfOrmer for Vision-Language Representation Learningcited by 45
  30. 2021Self-Distilled Self-Supervised Representation Learningcited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 6; 2024: 3; 2025: 6; 2026: 6.

21 papers.

Sort:

By area:

  1. 2023Vision-Language Models for Vision Tasks: A Surveycited by 834
  2. 2023EgoVLPv2: Egocentric Video-Language Pre-training with Fusion in the Backbonecited by 130
  3. 2023Fine-Grained Image-Text Alignment in Medical Imaging Enables Explainable Cyclic Image-Report Generationcited by 25
  4. 2023Reinforced UI Instruction Grounding: Towards a Generic UI Task Automation APIcited by 18
  5. 2024Fine-Grained Verifiers: Preference Modeling as Next-token Prediction in Vision-Language Alignmentcited by 17
  6. 2025Joint Fusion and Encoding: Advancing Multimodal Retrieval from the Ground Upcited by 6
  7. 2025LGD: Leveraging Generative Descriptions for Zero-Shot Referring Image Segmentationcited by 4
  8. 2026Physical Prompt Injection Attacks on Large Vision-Language Modelscited by 3
  9. 2026Gastric-X: A Multimodal Multi-Phase Benchmark Dataset for Advancing Vision-Language Models in Gastric Cancer Analysiscited by 2
  10. 2026Let ViT Speak: Generative Language-Image Pre-trainingcited by 2
  11. 2023Analyzing Multimodal Objectives Through the Lens of Generative Diffusion Guidancecited by 1
  12. 2024StochCA: A Novel Approach for Exploiting Pretrained Models with Cross-Attentioncited by 1
9 more
  1. 2025Mapping User Trust in Vision Language Models: Research Landscape, Challenges, and Prospectscited by 1
  2. 2025EvoVLMA: Evolutionary Vision-Language Model Adaptationcited by 1
  3. 2026Retrieval-Augmented Generation Based Nurse Observation Extractioncited by 1
  4. 2023ConcatPlexer: Additional Dim1 Batching for Faster ViTscited by 0
  5. 2024Fashion Style Editing with Generative Human Priorcited by 0
  6. 2025Understanding Differential Transformer Unchains Pretrained Self-Attentionscited by 0
  7. 2025Trust in Vision-Language Models: Insights from a Participatory User Workshopcited by 0
  8. 2026Disentangle Object and Non-object Infrared Features via Language Guidancecited by 0
  9. 2026Generative Universal Multimodal Retrieval with Dual-role Identifierscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2021Contrastive Vision-Language Pre-training with Limited Resources
  2. 2022ERNIE-ViL 2.0: Multi-view Contrastive Learning for Image-Text Pre-training
  3. 2021Probing Inter-modality: Visual Parsing with Self-Attention for Vision-Language Pre-training
  4. 2022UniCLIP: Unified Framework for Contrastive Language-Image Pre-training
  5. 2022Vision-Language Intelligence: Tasks, Representation Learning, and Large Models
  6. 2024Demonstrating and Reducing Shortcuts in Vision-Language Representation Learning
  7. 2025MMRL: Multi-Modal Representation Learning for Vision-Language Models
  8. 2022Augmenting Vision Language Pretraining by Learning Codebook with Visual Semantics