mlsanityData to 7 Oct 2026

Learning Transferable Visual Models From Natural Language Supervision

arXiv 2103.00020 ↗First posted 26 Feb 2021cs.CV cs.LGcited by 33,456

On the map: World models → Video-generation world models and simulators

Abstract (arXiv)

State-of-the-art computer vision systems are trained to predict a fixed set of predetermined object categories. This restricted form of supervision limits their generality and usability since additional labeled data is needed to specify any other visual concept. Learning directly from raw text about images is a promising alternative which leverages a much broader source of supervision. We demonstrate that the simple pre-training task of predicting which caption goes with which image is an efficient and scalable way to learn SOTA image representations from scratch on a dataset of 400 million (image, text) pairs collected from the internet. After pre-training, natural language is used to reference learned visual concepts (or describe new ones) enabling zero-shot transfer of the model to downstream tasks. We study the performance of this approach by benchmarking on over 30 different existing computer vision datasets, spanning tasks such as OCR, action recognition in videos, geo-localization, and many types of fine-grained object classification. The model transfers non-trivially to most tasks and is often competitive with a fully supervised baseline without the need for any dataset specific training. For instance, we match the accuracy of the original ResNet-50 on ImageNet zero-shot without needing to use any of the 1.28 million training examples it was trained on. We release our code and pre-trained model weights at https://github.com/OpenAI/CLIP.

Key facts. Cited by 33,456 papers in the map (data through 7 Oct 2026). Placed in Video-generation world models and simulators, one of 11 directions in World models. Builds on 135 of its 220 references that are in the map. First posted on arXiv 26 Feb 2021.

Builds on

135 of 220 references.

Closest ancestors · 1

Same direction, newest first

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale22 Oct

Other references · 134

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  5. 2020Language Models are Few-Shot Learnerscited by 32,199
  6. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  7. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  8. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  9. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  10. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  11. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  12. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
38 more
  1. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  2. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  3. 2020Scaling Laws for Neural Language Modelscited by 6,411
  4. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  5. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  6. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  7. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
  8. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,462
  9. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  10. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  11. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,857
  12. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  13. 2018Deep contextualized word representationscited by 2,285
  14. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,282
  15. 2013Visualizing and Understanding Convolutional Networkscited by 2,278
  16. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  17. 2020Shortcut Learning in Deep Neural Networkscited by 1,928
  18. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,842
  19. 2019Towards VQA Models That Can Readcited by 1,696
  20. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  21. 2016Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learningcited by 1,509
  22. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,507
  23. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,488
  24. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,445
  25. 2018ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustnesscited by 1,430
  26. 2015Spatial Transformer Networkscited by 1,423
  27. 2019Natural Adversarial Examplescited by 1,401
  28. 2020Big Self-Supervised Models are Strong Semi-Supervised Learnerscited by 1,375
  29. 2016Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddingscited by 1,304
  30. 2019Self-training with Noisy Student improves ImageNet classificationcited by 1,304
  31. 2019Contrastive Multiview Codingcited by 1,297
  32. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,255
  33. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  34. 2017Mixed Precision Trainingcited by 1,135
  35. 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,010
  36. 2016HyperNetworkscited by 1,003
  37. 2019Adversarial Examples Are Not Bugs, They Are Featurescited by 950
  38. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 946

50 most cited of 134 All 134 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 504; 2022: 1919; 2023: 5491; 2024: 8559; 2025: 9849; 2026: 7134.

33,456 papers.

Sort:

By area:

  1. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  2. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  3. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  4. 2023Segment Anythingcited by 8,158
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  6. 2023Visual Instruction Tuningcited by 7,792
  7. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  8. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  9. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  10. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  11. 2022Scalable Diffusion Models with Transformerscited by 5,931
  12. 2022Generative Adversarial Networkscited by 5,603
38 more
  1. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  2. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  3. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  4. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  5. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
  6. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  7. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  8. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,094
  9. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  10. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  11. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,286
  12. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  13. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,006
  14. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  15. 2021SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformerscited by 2,923
  16. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,777
  17. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,730
  18. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,591
  19. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  20. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,526
  21. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,409
  22. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,388
  23. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  24. 2024π0: A Vision-Language-Action Flow Model for General Robot Controlcited by 2,297
  25. 2021Learning to Prompt for Vision-Language Modelscited by 2,269
  26. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,231
  27. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  28. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  29. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  30. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
  31. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
  32. 2023VBench: Comprehensive Benchmark Suite for Video Generative Modelscited by 1,755
  33. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,725
  34. 2025Gemma 3 Technical Reportcited by 1,686
  35. 2022Visual Prompt Tuningcited by 1,674
  36. 2025π0.5: a Vision-Language-Action Model with Open-World Generalizationcited by 1,653
  37. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  38. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,610

50 most cited of 33,456 All 33,456 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023InstructCV: Instruction-Tuned Text-to-Image Diffusion Models as Vision Generalists
  2. 2022Vision-Language Pre-training: Basics, Recent Advances, and Future Trends
  3. 2026DV-SFT: Direct Vision Supervision for Fine-Grained Visual Understanding
  4. 2025Benchmarking Large Vision-Language Models on Fine-Grained Image Tasks: A Comprehensive Evaluation
  5. 2025Context Matters: Learning Global Semantics via Object-Centric Representation
  6. 2025Discovering Meaningful Units with Visually Grounded Semantics from Image Captions
  7. 2025Adapting Vision-Language Models Without Labels: A Comprehensive Survey
  8. 2026Benchmarking Large Vision-Language Models on Fine-Grained Image Tasks: From Evaluation to Diagnosis