mlsanityData to 14 Oct 2026

LiT: Zero-Shot Transfer with Locked-image text Tuning

arXiv 2111.07991 ↗First posted 15 Nov 2021cs.CV cs.CL cs.LGcited by 577

Not on the map. Why?

Cited by 203 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

This paper presents contrastive-tuning, a simple method employing contrastive training to align image and text models while still taking advantage of their pre-training. In our empirical study we find that locked pre-trained image models with unlocked text models work best. We call this instance of contrastive-tuning "Locked-image Tuning" (LiT), which just teaches a text model to read out good representations from a pre-trained image model for new tasks. A LiT model gains the capability of zero-shot transfer to new vision tasks, such as image classification or retrieval. The proposed LiT is widely applicable; it works reliably with multiple pre-training methods (supervised and unsupervised) and across diverse architectures (ResNet, Vision Transformers and MLP-Mixer) using three different image-text datasets. With the transformer-based pre-trained ViT-g/14 model, the LiT model achieves 85.2% zero-shot transfer accuracy on the ImageNet test set, and 82.5% on the challenging out-of-distribution ObjectNet test set.

Key facts. Cited by 577 papers in the map (data through 14 Oct 2026). Builds on 57 of its 73 references that are in the map. First posted on arXiv 15 Nov 2021.

Builds on

57 of 73 references.

References · 57

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  5. 2020Language Models are Few-Shot Learnerscited by 32,419
  6. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  7. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  8. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  9. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  10. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  11. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  12. 2014Going Deeper with Convolutionscited by 4,241
38 more
  1. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  2. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  3. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,222
  4. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,989
  5. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,850
  6. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,818
  7. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,616
  8. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,515
  9. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,475
  10. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,462
  11. 2021MLP-Mixer: An all-MLP Architecture for Visioncited by 1,446
  12. 2019Natural Adversarial Examplescited by 1,408
  13. 2016Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzlescited by 1,382
  14. 2020Big Self-Supervised Models are Strong Semi-Supervised Learnerscited by 1,380
  15. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,296
  16. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,216
  17. 2015Unsupervised Visual Representation Learning by Context Predictioncited by 1,066
  18. 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,013
  19. 2021Scaling Vision Transformerscited by 999
  20. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 968
  21. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  22. 2017Revisiting Unreasonable Effectiveness of Data in Deep Learning Eracited by 919
  23. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 905
  24. 2015Improving Neural Machine Translation Models with Monolingual Datacited by 788
  25. 2019Big Transfer (BiT): General Visual Representation Learningcited by 769
  26. 2020Contrastive Learning of Medical Visual Representations from Paired Images and Textcited by 641
  27. 2018Exploring the Limits of Weakly Supervised Pretrainingcited by 605
  28. 2021CoAtNet: Marrying Convolution and Attention for All Data Sizescited by 569
  29. 2021How to train your ViT? Data, Augmentation, and Regularization in Vision Transformerscited by 569
  30. 2018Do Better ImageNet Models Transfer Better?cited by 565
  31. 2018From Recognition to Cognition: Visual Commonsense Reasoningcited by 556
  32. 2017Zero-Shot Learning -- A Comprehensive Evaluation of the Good, the Bad and the Uglycited by 538
  33. 2013DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognitioncited by 424
  34. 2018A Survey on Deep Transfer Learningcited by 419
  35. 2020VirTex: Learning Visual Representations from Textual Annotationscited by 339
  36. 2020Are we done with ImageNet?cited by 333
  37. 2014CNN Features off-the-shelf: an Astounding Baseline for Recognitioncited by 316
  38. 2021WIT: Wikipedia-based Image Text Dataset for Multimodal Multilingual Machine Learningcited by 287

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 2; 2022: 73; 2023: 178; 2024: 147; 2025: 103; 2026: 74.

577 papers.

Sort:

By area:

  1. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  2. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  3. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  4. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  5. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
  6. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,224
  7. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,274
  8. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,175
  9. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,173
  10. 2023Human Preference Score v2: A Solid Benchmark for Evaluating Human Preferences of Text-to-Image Synthesiscited by 890
  11. 2023Vision-Language Models for Vision Tasks: A Surveycited by 839
  12. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 781
38 more
  1. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 766
  2. 2023Scaling Vision Transformers to 22 Billion Parameterscited by 694
  3. 2022MaPLe: Multi-modal Prompt Learningcited by 687
  4. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Modelscited by 668
  5. 2022When and why vision-language models behave like bags-of-words, and what to do about it?cited by 610
  6. 2021Zero-Shot Text-Guided Object Generation with Dream Fieldscited by 513
  7. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
  8. 2023Symbolic Discovery of Optimization Algorithmscited by 434
  9. 2022Test-Time Prompt Tuning for Zero-Shot Generalization in Vision-Language Modelscited by 382
  10. 2023Scaling Open-Vocabulary Object Detectioncited by 343
  11. 2022Simple Open-Vocabulary Object Detection with Vision Transformerscited by 305
  12. 2022Visual Spatial Reasoningcited by 293
  13. 2023EmbodiedGPT: Vision-Language Pre-Training via Embodied Chain of Thoughtcited by 279
  14. 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 279
  15. 2023Self-regulating Prompts: Foundational Model Adaptation without Forgettingcited by 272
  16. 2023A Cookbook of Self-Supervised Learningcited by 250
  17. 2023Data Filtering Networkscited by 250
  18. 2023From Sparse to Soft Mixtures of Expertscited by 248
  19. 2023SugarCrepe: Fixing Hackable Benchmarks for Vision-Language Compositionalitycited by 243
  20. 2023Improving CLIP Training with Language Rewritescited by 217
  21. 2022Multimodal Contrastive Learning with LIMoE: the Language-Image Mixture of Expertscited by 203
  22. 2023Teaching CLIP to Count to Tencited by 198
  23. 2022CoWs on Pasture: Baselines and Benchmarks for Language-Driven Zero-Shot Object Navigationcited by 194
  24. 2022Patching open-vocabulary models by interpolating weightscited by 190
  25. 2021Combined Scaling for Zero-shot Transfer Learningcited by 182
  26. 2023Reconstructing the Mind's Eye: fMRI-to-Image with Contrastive Learning and Diffusion Priorscited by 160
  27. 2022FlexiViT: One Model for All Patch Sizescited by 152
  28. 2022F-VLM: Open-Vocabulary Object Detection upon Frozen Vision and Language Modelscited by 150
  29. 2022CLIP-Dissect: Automatic Description of Neuron Representations in Deep Vision Networkscited by 147
  30. 2022Linearly Mapping from Image to Text Spacecited by 142
  31. 2022Data Determines Distributional Robustness in Contrastive Language Image Pre-training (CLIP)cited by 138
  32. 2023Visual Language Pretrained Multiple Instance Zero-Shot Transfer for Histopathology Imagescited by 134
  33. 2024Exploring scalable medical image encoders beyond text supervisioncited by 132
  34. 2023EgoVLPv2: Egocentric Video-Language Pre-training with Fusion in the Backbonecited by 130
  35. 2023PaLI-3 Vision Language Models: Smaller, Faster, Strongercited by 126
  36. 2022REVEAL: Retrieval-Augmented Visual-Language Pre-Training with Multi-Source Multimodal Knowledge Memorycited by 123
  37. 2023Aligning Bag of Regions for Open-Vocabulary Object Detectioncited by 119
  38. 2023ONE-PEACE: Exploring One General Representation Model Toward Unlimited Modalitiescited by 118

50 most cited of 577 All 577 (JSON)