mlsanityData to 14 Oct 2026

Multimodal Few-Shot Learning with Frozen Language Models

arXiv 2106.13884 ↗First posted 25 Jun 2021cs.CV cs.CL cs.LGcited by 661

Not on the map. Why?

Cited by 210 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

When trained at sufficient scale, auto-regressive language models exhibit the notable ability to learn a new language task after being prompted with just a few examples. Here, we present a simple, yet effective, approach for transferring this few-shot learning ability to a multimodal setting (vision and language). Using aligned image and caption data, we train a vision encoder to represent each image as a sequence of continuous embeddings, such that a pre-trained, frozen language model prompted with this prefix generates the appropriate caption. The resulting system is a multimodal few-shot learner, with the surprising ability to learn a variety of new tasks when conditioned on examples, represented as a sequence of multiple interleaved image and text embeddings. We demonstrate that it can rapidly learn words for new objects and novel visual categories, do visual question-answering with only a handful of examples, and make use of outside knowledge, by measuring a single model on a variety of established and new benchmarks.

Key facts. Cited by 661 papers in the map (data through 14 Oct 2026). Builds on 36 of its 46 references that are in the map. First posted on arXiv 25 Jun 2021.

Builds on

36 of 46 references.

References · 36

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2020Language Models are Few-Shot Learnerscited by 32,419
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  6. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  7. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  8. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  9. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  10. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  11. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
  12. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
24 more
  1. 2016Matching Networks for One Shot Learningcited by 2,255
  2. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,222
  3. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,141
  4. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,989
  5. 2019Language Models as Knowledge Bases?cited by 1,936
  6. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  7. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,214
  8. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,121
  9. 2018Self-Attention with Relative Position Representationscited by 1,120
  10. 2021Scaling Vision Transformerscited by 999
  11. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  12. 2020How Much Knowledge Can You Pack Into the Parameters of a Language Model?cited by 719
  13. 2017In-Datacenter Performance Analysis of a Tensor Processing Unitcited by 618
  14. 2020Towards a Human-like Open-Domain Chatbotcited by 470
  15. 2021Unifying Vision-and-Language Tasks via Text Generationcited by 412
  16. 2021High-Performance Large-Scale Image Recognition Without Normalizationcited by 360
  17. 2019Rapid Learning or Feature Reuse? Towards Understanding the Effectiveness of MAMLcited by 359
  18. 2018A Comprehensive Survey of Deep Learning for Image Captioningcited by 214
  19. 2021VisualGPT: Data-efficient Adaptation of Pretrained Language Models for Image Captioningcited by 191
  20. 2021Pretrained Transformers as Universal Computation Enginescited by 162
  21. 2019Supervised Multimodal Bitransformers for Classifying Images and Textcited by 130
  22. 2021Multi-Modal Answer Validation for Knowledge-Based VQAcited by 79
  23. 2019Encoder-Agnostic Adaptation for Conditional Language Generationcited by 27
  24. 2019Latent Translation: Crossing Modalities by Bridging Generative Modelscited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 16; 2022: 105; 2023: 226; 2024: 173; 2025: 89; 2026: 52.

661 papers.

Sort:

By area:

  1. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  2. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  3. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  4. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  5. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,245
  6. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,234
  7. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,071
  8. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,471
  9. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,439
  10. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,274
  11. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,239
  12. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,035
38 more
  1. 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapterscited by 1,026
  2. 2023A Survey on Multimodal Large Language Modelscited by 909
  3. 2022A Generalist Agentcited by 808
  4. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 781
  5. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 727
  6. 2022A Survey on In-context Learningcited by 721
  7. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 708
  8. 2023Time-LLM: Time Series Forecasting by Reprogramming Large Language Modelscited by 700
  9. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Modelscited by 668
  10. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 647
  11. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 625
  12. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  13. 2025SmolVLA: A Vision-Language-Action Model for Affordable and Efficient Roboticscited by 530
  14. 2023MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Actioncited by 517
  15. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
  16. 2021Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Surveycited by 463
  17. 2022VIMA: General Robot Manipulation with Multimodal Promptscited by 407
  18. 2022Multimodal Learning with Transformers: A Surveycited by 400
  19. 2023Large Language Models for Software Engineering: A Systematic Literature Reviewcited by 399
  20. 2023An Embodied Generalist Agent in 3D Worldcited by 365
  21. 2024Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Modelscited by 329
  22. 2021VL-Adapter: Parameter-Efficient Transfer Learning for Vision-and-Language Taskscited by 320
  23. 2023Visual-Language Prompt Tuning with Knowledge-guided Context Optimizationcited by 308
  24. 2023Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Actioncited by 302
  25. 2021An Empirical Study of GPT-3 for Few-Shot Knowledge-Based VQAcited by 301
  26. 2023A Survey on Multimodal Large Language Models for Autonomous Drivingcited by 290
  27. 2023Towards Generalist Biomedical AIcited by 289
  28. 2022VQGAN-CLIP: Open Domain Image Generation and Editing with Natural Language Guidancecited by 285
  29. 2023On Evaluating Adversarial Robustness of Large Vision-Language Modelscited by 284
  30. 2023EmbodiedGPT: Vision-Language Pre-Training via Embodied Chain of Thoughtcited by 279
  31. 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 279
  32. 2022Exploring Visual Prompts for Adapting Large-Scale Modelscited by 274
  33. 2023Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioningcited by 263
  34. 2023Generating Images with Multimodal Language Modelscited by 260
  35. 2024LogicVista: Multimodal LLM Logical Reasoning Benchmark in Visual Contextscited by 245
  36. 2024MM1: Methods, Analysis & Insights from Multimodal LLM Pre-trainingcited by 232
  37. 2022Zero-Shot Video Question Answering via Frozen Bidirectional Language Modelscited by 229
  38. 2023A Comprehensive Survey of AI-Generated Content (AIGC): A History of Generative AI from GAN to ChatGPTcited by 224

50 most cited of 661 All 661 (JSON)