mlsanityData to 7 Oct 2026

How Much Can CLIP Benefit Vision-and-Language Tasks?

arXiv 2107.06383 ↗First posted 13 Jul 2021cs.CV cs.AI cs.CLcited by 321

Not on the map. Why?

Cited by 73 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Most existing Vision-and-Language (V&L) models rely on pre-trained visual encoders, using a relatively small set of manually-annotated data (as compared to web-crawled data), to perceive the visual world. However, it has been observed that large-scale pretraining usually can result in better generalization performance, e.g., CLIP (Contrastive Language-Image Pre-training), trained on a massive amount of image-caption pairs, has shown a strong zero-shot capability on various vision tasks. To further study the advantage brought by CLIP, we propose to use CLIP as the visual encoder in various V&L models in two typical scenarios: 1) plugging CLIP into task-specific fine-tuning; 2) combining CLIP with V&L pre-training and transferring to downstream tasks. We show that CLIP significantly outperforms widely-used visual encoders trained with in-domain annotated data, such as BottomUp-TopDown. We achieve competitive or better results on diverse V&L tasks, while establishing new state-of-the-art results on Visual Question Answering, Visual Entailment, and V&L Navigation tasks. We release our code at https://github.com/clip-vil/CLIP-ViL.

Key facts. Cited by 321 papers in the map (data through 7 Oct 2026). Builds on 66 of its 89 references that are in the map. First posted on arXiv 13 Jul 2021.

Builds on

66 of 89 references.

References · 66

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  7. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  8. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,690
  9. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  10. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  11. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  12. 2015VQA: Visual Question Answeringcited by 2,489
38 more
  1. 2015Learning Deep Features for Discriminative Localizationcited by 2,430
  2. 2014Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modelingcited by 2,341
  3. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  4. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  5. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  6. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  7. 2017Matterport3D: Learning from RGB-D Data in Indoor Environmentscited by 1,354
  8. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,310
  9. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  10. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  11. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  12. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,013
  13. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 966
  14. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  15. 2016SPICE: Semantic Propositional Image Caption Evaluationcited by 820
  16. 2020Stanza: A Python Natural Language Processing Toolkit for Many Human Languagescited by 719
  17. 2016Self-critical Sequence Training for Image Captioningcited by 544
  18. 2019Unified Vision-Language Pre-Training for Image Captioning and VQAcited by 494
  19. 2020Beyond the Nav-Graph: Vision-and-Language Navigation in Continuous Environmentscited by 429
  20. 2020Room-Across-Room: Multilingual Vision-and-Language Navigation with Dense Spatiotemporal Groundingcited by 401
  21. 2019Deep Modular Co-Attention Networks for Visual Question Answeringcited by 338
  22. 2020VirTex: Learning Visual Representations from Textual Annotationscited by 338
  23. 2018Speaker-Follower Models for Vision-and-Language Navigationcited by 337
  24. 2019REVERIE: Remote Embodied Visual Referring Expression in Real Indoor Environmentscited by 330
  25. 2015Visual7W: Grounded Question Answering in Imagescited by 315
  26. 2018Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigationcited by 304
  27. 2020A Recurrent Vision-and-Language BERT for Navigationcited by 300
  28. 2019Attention on Attention for Image Captioningcited by 289
  29. 2020Pixel-BERT: Aligning Image Pixels with Text by Deep Multi-Modal Transformerscited by 288
  30. 2015Stacked Attention Networks for Image Question Answeringcited by 273
  31. 2019Visual Entailment: A Novel Task for Fine-Grained Image Understandingcited by 250
  32. 2016Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Groundingcited by 243
  33. 2019Learning to Navigate Unseen Environments: Back Translation with Environmental Dropoutcited by 241
  34. 2018Touchdown: Natural Language Navigation and Spatial Reasoning in Visual Street Environmentscited by 238
  35. 2020Towards Learning a Generic Agent for Vision-and-Language Navigation via Pre-trainingcited by 228
  36. 2019Vision-Language Navigation with Self-Supervised Auxiliary Reasoning Taskscited by 177
  37. 2019Self-Monitoring Navigation Agent via Auxiliary Progress Estimationcited by 173
  38. 2020In Defense of Grid Features for Visual Question Answeringcited by 155

50 most cited of 66 All 66 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 22; 2022: 90; 2023: 89; 2024: 61; 2025: 39; 2026: 20.

321 papers.

Sort:

By area:

  1. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  2. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  3. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
  4. 2023VideoChat: Chat-Centric Video Understandingcited by 851
  5. 2021Florence: A New Foundation Model for Computer Visioncited by 802
  6. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 645
  7. 2024GPT-4V(ision) is a Generalist Web Agent, if Groundedcited by 547
  8. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 536
  9. 2023InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generationcited by 488
  10. 2022InternVideo: General Video Foundation Models via Generative and Discriminative Learningcited by 441
  11. 2021Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigmcited by 405
  12. 2023RemoteCLIP: A Vision Language Foundation Model for Remote Sensingcited by 357
38 more
  1. 2021VL-Adapter: Parameter-Efficient Transfer Learning for Vision-and-Language Taskscited by 319
  2. 2021An Empirical Study of Training End-to-End Vision-and-Language Transformerscited by 271
  3. 2021CLIP-Forge: Towards Zero-Shot Text-to-Shape Generationcited by 254
  4. 2023A Cookbook of Self-Supervised Learningcited by 248
  5. 2022Zero-Shot Video Question Answering via Frozen Bidirectional Language Modelscited by 229
  6. 2021Simple but Effective: CLIP Embeddings for Embodied AIcited by 191
  7. 2022Patching open-vocabulary models by interpolating weightscited by 187
  8. 2022mPLUG: Effective and Efficient Vision-Language Learning by Cross-modal Skip-connectionscited by 185
  9. 2022CREPE: Can Vision-Language Foundation Models Reason Compositionally?cited by 177
  10. 2023SCLIP: Rethinking Self-Attention for Dense Vision-Language Inferencecited by 169
  11. 2022Revisiting the "Video" in Video-Language Understandingcited by 167
  12. 2022VLP: A Survey on Vision-Language Pre-trainingcited by 159
  13. 2023Large-scale Multi-Modal Pre-trained Models: A Comprehensive Surveycited by 153
  14. 2021ZeroCap: Zero-Shot Image-to-Text Generation for Visual-Semantic Arithmeticcited by 150
  15. 2022Linearly Mapping from Image to Text Spacecited by 139
  16. 2021From Show to Tell: A Survey on Deep Learning-based Image Captioningcited by 137
  17. 2022PromptFL: Let Federated Participants Cooperatively Learn Prompts Instead of Models -- Federated Learning in Age of Foundation Modelcited by 133
  18. 2022Open-Vocabulary Universal Image Segmentation with MaskCLIPcited by 131
  19. 2023Knowledge-enhanced Visual-Language Pre-training on Chest Radiology Imagescited by 128
  20. 2023ESC: Exploration with Soft Commonsense Constraints for Zero-shot Object Navigationcited by 126
  21. 2021UniTAB: Unifying Text and Box Outputs for Grounded Vision-Language Modelingcited by 122
  22. 2022ReCLIP: A Strong Zero-Shot Baseline for Referring Expression Comprehensioncited by 119
  23. 2022BEVBert: Multimodal Map Pre-training for Language-guided Navigationcited by 117
  24. 2022UniFormerV2: Spatiotemporal Learning by Arming Image ViTs with Video UniFormercited by 116
  25. 2023Scaling Data Generation in Vision-and-Language Navigationcited by 115
  26. 2022CLIP Models are Few-shot Learners: Empirical Studies on VQA and Visual Entailmentcited by 113
  27. 2022Coarse-to-Fine Vision-Language Pre-training with Fusion in the Backbonecited by 109
  28. 2022Chinese CLIP: Contrastive Vision-Language Pretraining in Chinesecited by 108
  29. 2023Evaluation and Analysis of Hallucination in Large Vision-Language Modelscited by 99
  30. 2022Semantic Abstraction: Open-World 3D Scene Understanding from 2D Vision-Language Modelscited by 95
  31. 2022Text-Only Training for Image Captioning using Noise-Injected CLIPcited by 91
  32. 2022Language Models Can See: Plugging Visual Controls in Text Generationcited by 89
  33. 2023Image Captioners Are Scalable Vision Learners Toocited by 89
  34. 2021CLOOB: Modern Hopfield Networks with InfoLOOB Outperform CLIPcited by 88
  35. 2023Scaling Vision-Language Models with Sparse Mixture of Expertscited by 88
  36. 2022Reliable Visual Question Answering: Abstain Rather Than Answer Incorrectlycited by 86
  37. 2022CenterCLIP: Token Clustering for Efficient Text-Video Retrievalcited by 86
  38. 2021MAGMA -- Multimodal Augmentation of Generative Models through Adapter-based Finetuningcited by 85

50 most cited of 321 All 321 (JSON)