mlsanityData to 14 Oct 2026

Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision

arXiv 2102.05918 ↗First posted 11 Feb 2021cs.CV cs.CL cs.LGcited by 3,864

Not on the map. Why?

Cited by 1,213 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Pre-trained representations are becoming crucial for many NLP and perception tasks. While representation learning in NLP has transitioned to training on raw text without human annotations, visual and vision-language representations still rely heavily on curated training datasets that are expensive or require expert knowledge. For vision applications, representations are mostly learned using datasets with explicit class labels such as ImageNet or OpenImages. For vision-language, popular datasets like Conceptual Captions, MSCOCO, or CLIP all involve a non-trivial data collection (and cleaning) process. This costly curation process limits the size of datasets and hence hinders the scaling of trained models. In this paper, we leverage a noisy dataset of over one billion image alt-text pairs, obtained without expensive filtering or post-processing steps in the Conceptual Captions dataset. A simple dual-encoder architecture learns to align visual and language representations of the image and text pairs using a contrastive loss. We show that the scale of our corpus can make up for its noise and leads to state-of-the-art representations even with such a simple learning scheme. Our visual representation achieves strong performance when transferred to classification tasks such as ImageNet and VTAB. The aligned visual and language representations enables zero-shot image classification and also set new state-of-the-art results on Flickr30K and MSCOCO image-text retrieval benchmarks, even when compared with more sophisticated cross-attention models. The representations also enable cross-modality search with complex text and text + image queries.

Key facts. Cited by 3,864 papers in the map (data through 14 Oct 2026). Builds on 59 of its 81 references that are in the map. First posted on arXiv 11 Feb 2021.

Builds on

59 of 81 references.

References · 59

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  5. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  6. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  7. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  8. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,375
  9. 2014Going Deeper with Convolutionscited by 4,241
  10. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,474
  11. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,857
  12. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,811
38 more
  1. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
  2. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,222
  3. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,850
  4. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,515
  5. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,462
  6. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,417
  7. 2019Natural Adversarial Examplescited by 1,408
  8. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,362
  9. 2019Self-training with Noisy Student improves ImageNet classificationcited by 1,311
  10. 2019Contrastive Multiview Codingcited by 1,301
  11. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,214
  12. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 968
  13. 2017Revisiting Unreasonable Effectiveness of Data in Deep Learning Eracited by 919
  14. 2019Self-Supervised Learning of Pretext-Invariant Representationscited by 808
  15. 2019Big Transfer (BiT): General Visual Representation Learningcited by 769
  16. 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 674
  17. 2019Data-Efficient Image Recognition with Contrastive Predictive Codingcited by 673
  18. 2020Contrastive Learning of Medical Visual Representations from Paired Images and Textcited by 641
  19. 2020Prototypical Contrastive Learning of Unsupervised Representationscited by 640
  20. 2018Exploring the Limits of Weakly Supervised Pretrainingcited by 605
  21. 2019A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmarkcited by 425
  22. 2020Meta Pseudo Labelscited by 374
  23. 2020Rethinking Pre-training and Self-trainingcited by 352
  24. 2020VirTex: Learning Visual Representations from Textual Annotationscited by 339
  25. 2020Large-Scale Adversarial Training for Vision-and-Language Representation Learningcited by 327
  26. 2020A Metric Learning Reality Checkcited by 321
  27. 2020Pixel-BERT: Aligning Image Pixels with Text by Deep Multi-Modal Transformerscited by 288
  28. 2016Multi30K: Multilingual English-German Image Descriptionscited by 250
  29. 2019Billion-scale semi-supervised learning for image classificationcited by 232
  30. 2020ERNIE-ViL: Knowledge Enhanced Vision-Language Representations Through Scene Graphcited by 211
  31. 2019Fixing the train-test resolution discrepancycited by 209
  32. 2014Unifying Visual-Semantic Embeddings with Multimodal Neural Language Modelscited by 198
  33. 2014Learning Fine-grained Image Similarity with Deep Rankingcited by 190
  34. 2019Visual Semantic Reasoning for Image-Text Matchingcited by 185
  35. 2014SimLex-999: Evaluating Semantic Models with (Genuine) Similarity Estimationcited by 172
  36. 2015Learning Visual Features from Large Weakly Supervised Datacited by 165
  37. 2020ImageBERT: Cross-modal Pre-training with Large-scale Weak-supervised Image-Text Datacited by 137
  38. 2020Learning the Best Pooling Strategy for Visual Semantic Embeddingcited by 137

50 most cited of 59 All 59 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 107; 2022: 461; 2023: 968; 2024: 955; 2025: 833; 2026: 540.

3,864 papers.

Sort:

By area:

  1. 2023Segment Anythingcited by 8,234
  2. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  3. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  4. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  5. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  6. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  7. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  8. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
  9. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,424
  10. 2021Learning to Prompt for Vision-Language Modelscited by 2,281
  11. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,224
  12. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
38 more
  1. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,619
  2. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,555
  3. 2022Objaverse: A Universe of Annotated 3D Objectscited by 1,540
  4. 2021MLP-Mixer: An all-MLP Architecture for Visioncited by 1,446
  5. 2022Conditional Prompt Learning for Vision-Language Modelscited by 1,433
  6. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,396
  7. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,295
  8. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,274
  9. 2022Text Embeddings by Weakly-Supervised Contrastive Pre-trainingcited by 1,253
  10. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,177
  11. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,175
  12. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,173
  13. 2021Grounded Language-Image Pre-trainingcited by 1,156
  14. 2022Editing Models with Task Arithmeticcited by 1,120
  15. 2024Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Modelcited by 1,056
  16. 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapterscited by 1,026
  17. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,005
  18. 2021Scaling Vision Transformerscited by 999
  19. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 914
  20. 2021Open-vocabulary Object Detection via Vision and Language Knowledge Distillationcited by 906
  21. 2023Vision-Language Models for Vision Tasks: A Surveycited by 839
  22. 2022A Generalist Agentcited by 808
  23. 2021Florence: A New Foundation Model for Computer Visioncited by 804
  24. 2021Robust fine-tuning of zero-shot modelscited by 797
  25. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 781
  26. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 766
  27. 2023Scaling Vision Transformers to 22 Billion Parameterscited by 694
  28. 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 691
  29. 2022MaPLe: Multi-modal Prompt Learningcited by 687
  30. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 661
  31. 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 656
  32. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 649
  33. 2023Muse: Text-To-Image Generation via Masked Generative Transformerscited by 648
  34. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 647
  35. 2023DataComp: In search of the next generation of multimodal datasetscited by 644
  36. 2021FILIP: Fine-grained Interactive Language-Image Pre-Trainingcited by 627
  37. 2021RegionCLIP: Region-based Language-Image Pretrainingcited by 616
  38. 2022When and why vision-language models behave like bags-of-words, and what to do about it?cited by 610

50 most cited of 3,864 All 3,864 (JSON)