mlsanityData to 7 Oct 2026

Diversifying Joint Vision-Language Tokenization Learning

arXiv 2306.03421 ↗First posted 6 Jun 2023cs.CVcited by 0

Not on the map. Why?

Abstract (arXiv)

Building joint representations across images and text is an essential step for tasks such as Visual Question Answering and Video Question Answering. In this work, we find that the representations must not only jointly capture features from both modalities but should also be diverse for better generalization performance. To this end, we propose joint vision-language representation learning by diversifying the tokenization learning process, enabling tokens that are sufficiently disentangled from each other to be learned from both modalities. We observe that our approach outperforms the baseline models in a majority of settings and is competitive with state-of-the-art methods.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 26 of its 37 references that are in the map. First posted on arXiv 6 Jun 2023.

Builds on

26 of 37 references.

References · 26

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  7. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  8. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  9. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  10. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  11. 2019SuperGlue: Learning Feature Matching with Graph Neural Networkscited by 1,437
  12. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
14 more
  1. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  2. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  3. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  4. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 946
  5. 2021Florence: A New Foundation Model for Computer Visioncited by 802
  6. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 645
  7. 2020X3D: Expanding Architectures for Efficient Video Recognitioncited by 635
  8. 2019Visual Entailment: A Novel Task for Fine-Grained Image Understandingcited by 250
  9. 2020Just Ask: Learning to Answer Questions from Millions of Narrated Videoscited by 248
  10. 2017DARLA: Improving Zero-Shot Transfer in Reinforcement Learningcited by 147
  11. 2021Improving Zero-shot Voice Style Transfer via Disentangled Representation Learningcited by 32
  12. 2021Cross-Domain and Disentangled Face Manipulation with 3D Guidancecited by 17
  13. 2022Video Question Answering with Iterative Video-Text Co-Tokenizationcited by 16
  14. 2021DAIR: Disentangled Attention Intrinsic Regularization for Safe and Efficient Bimanual Manipulationcited by 3

Built on by

Not cited yet.