mlsanityData to 7 Oct 2026

Is Multimodal Vision Supervision Beneficial to Language?

arXiv 2302.05016 ↗First posted 10 Feb 2023cs.CV cs.AI cs.CLcited by 6

Not on the map. Why?

Abstract (arXiv)

Vision (image and video) - Language (VL) pre-training is the recent popular paradigm that achieved state-of-the-art results on multi-modal tasks like image-retrieval, video-retrieval, visual question answering etc. These models are trained in an unsupervised way and greatly benefit from the complementary modality supervision. In this paper, we explore if the language representations trained using vision supervision perform better than vanilla language representations on Natural Language Understanding and commonsense reasoning benchmarks. We experiment with a diverse set of image-text models such as ALBEF, BLIP, METER and video-text models like ALPRO, Frozen-in-Time (FiT), VIOLET. We compare the performance of language representations of stand-alone text encoders of these models to the language representations of text encoders learnt through vision supervision. Our experiments suggest that vanilla language representations show superior performance on most of the tasks. These results shed light on the current drawbacks of the vision-language models.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 27 of its 36 references that are in the map. First posted on arXiv 10 Feb 2023.

Builds on

27 of 36 references.

References · 27

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  6. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  7. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  8. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  9. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  10. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  11. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  12. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
15 more
  1. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  2. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  3. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,175
  4. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 646
  5. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 645
  6. 2021MERLOT: Multimodal Neural Script Knowledge Modelscited by 311
  7. 2019Cosmos QA: Machine Reading Comprehension with Contextual Commonsense Reasoningcited by 281
  8. 2021An Empirical Study of Training End-to-End Vision-and-Language Transformerscited by 271
  9. 2019WinoGrande: An Adversarial Winograd Schema Challenge at Scalecited by 194
  10. 2021VIOLET : End-to-End Video-Language Transformers with Masked Visual-token Modelingcited by 187
  11. 2021Align and Prompt: Video-and-Language Pre-training with Entity Promptscited by 149
  12. 2021Effect of Visual Extensions on Natural Language Understanding in Vision-and-Language Modelscited by 18
  13. 2022Learning to Retrieve Videos by Asking Questionscited by 17
  14. 2019Sequential Learning of Convolutional Features for Effective Text Classificationcited by 3
  15. 2022MuMUR : Multilingual Multimodal Universal Retrievalcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 2; 2026: 1.

6 papers.

  1. 2025Vision-and-Language Training Helps Deploy Taxonomic Knowledge but Does Not Fundamentally Alter Itcited by 6
  2. 2024Acquiring Linguistic Knowledge from Multimodal Inputcited by 3
  3. 2024Quantifying and Enabling the Interpretability of CLIP-like Modelscited by 3
  4. 2024Vision-Language Models Align with Human Neural Representations in Concept Processingcited by 2
  5. 2025Pruning the Paradox: How CLIP's Most Informative Heads Enhance Performance While Amplifying Biascited by 1
  6. 2026Grounded Concreteness: Human-Like Concreteness Sensitivity in Vision-Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2022Training Vision-Language Models with Less Bimodal Supervision
  2. 2022A Survey of Vision-Language Pre-Trained Models
  3. 2023Stop Pre-Training: Adapt Visual-Language Models to Unseen Languages
  4. 2022Vision-Language Pre-training: Basics, Recent Advances, and Future Trends
  5. 2022Vision-Language Intelligence: Tasks, Representation Learning, and Large Models
  6. 2025Towards General Continuous Memory for Vision-Language Models
  7. 2025Seeing Beyond Words: Self-Supervised Visual Learning for Multimodal Large Language Models
  8. 2024Exploring the Frontier of Vision-Language Models: A Survey of Current Methodologies and Future Directions