mlsanityData to 7 Oct 2026

Does language help generalization in vision models?

arXiv 2104.08313 ↗First posted 16 Apr 2021cs.AI cs.CL cs.CVcited by 19

Not on the map. Why?

Abstract (arXiv)

Vision models trained on multimodal datasets can benefit from the wide availability of large image-caption datasets. A recent model (CLIP) was found to generalize well in zero-shot and transfer learning settings. This could imply that linguistic or "semantic grounding" confers additional generalization abilities to the visual feature space. Here, we systematically evaluate various multimodal architectures and vision-only models in terms of unsupervised clustering, few-shot learning, transfer learning and adversarial robustness. In each setting, multimodal training produced no additional generalization capability compared to standard supervised visual training. We conclude that work is still required for semantic grounding to help improve vision models.

Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). Builds on 18 of its 40 references that are in the map. First posted on arXiv 16 Apr 2021.

Builds on

18 of 40 references.

References · 18

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  6. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,346
  7. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,283
  8. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,462
  9. 2018ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustnesscited by 1,430
  10. 2019Adversarial Examples Are Not Bugs, They Are Featurescited by 950
  11. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 946
  12. 2019Big Transfer (BiT): General Visual Representation Learningcited by 766
6 more
  1. 2020VirTex: Learning Visual Representations from Textual Annotationscited by 338
  2. 2020Do Adversarially Robust ImageNet Models Transfer Better?cited by 315
  3. 2020Self-Supervised MultiModal Versatile Networkscited by 276
  4. 2015Learning Visual Features from Large Weakly Supervised Datacited by 165
  5. 2018Found in Translation: Learning Robust Joint Representations by Cyclic Translations Between Modalitiescited by 149
  6. 2020Learning Visual Representations with Caption Annotationscited by 123

Built on by

Cited by year

Citing papers per year: 2021: 1; 2022: 3; 2023: 6; 2024: 8; 2025: 1.

19 papers.

Sort:

By area:

  1. 2022Data Determines Distributional Robustness in Contrastive Language Image Pre-training (CLIP)cited by 137
  2. 2021CLOOB: Modern Hopfield Networks with InfoLOOB Outperform CLIPcited by 88
  3. 2023FLIP: Cross-domain Face Anti-spoofing with Language Guidancecited by 45
  4. 2022Is a Caption Worth a Thousand Images? A Controlled Study for Representation Learningcited by 39
  5. 2024Few-Shot Adversarial Prompt Learning on Vision-Language Modelscited by 39
  6. 2022Multi-Prompt Alignment for Multi-Source Unsupervised Domain Adaptationcited by 25
  7. 2023Language-Driven Anchors for Zero-Shot Adversarial Robustnesscited by 23
  8. 2023Effective Robustness against Natural Distribution Shifts for Models with Different Training Datacited by 13
  9. 2024What Makes CLIP More Robust to Long-Tailed Pre-Training Data? A Controlled Study for Transferable Insightscited by 13
  10. 2023Semi-supervised Multimodal Representation Learning through a Global Workspacecited by 11
  11. 2023Interpreting CLIP: Insights on the Robustness to ImageNet Distribution Shiftscited by 6
  12. 2024Tell, Don't Show!: Language Guidance Eases Transfer Across Domains in Images and Videoscited by 4
7 more
  1. 2023CapText: Large Language Model-based Caption Generation From Image Context and Descriptioncited by 3
  2. 2024Zero-shot cross-modal transfer of Reinforcement Learning policies through a Global Workspacecited by 3
  3. 2025Multimodal Dreaming: A Global Workspace Approach to World Model-Based Reinforcement Learningcited by 3
  4. 2024Multi-Task Domain Adaptation for Language Grounding with 3D Objectscited by 2
  5. 2024Contrastive Abstraction for Reinforcement Learningcited by 2
  6. 2024Adaptive Prompt Learning with Negative Textual Semantics and Uncertainty Modeling for Universal Multi-Source Domain Adaptationcited by 1
  7. 2024Semantically Guided Action Anticipationcited by 0