mlsanityData to 7 Oct 2026

BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation

arXiv 2201.12086 ↗First posted 28 Jan 2022cs.CVcited by 4,310

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Vision-Language Pre-training (VLP) has advanced the performance for many vision-language tasks. However, most existing pre-trained models only excel in either understanding-based tasks or generation-based tasks. Furthermore, performance improvement has been largely achieved by scaling up the dataset with noisy image-text pairs collected from the web, which is a suboptimal source of supervision. In this paper, we propose BLIP, a new VLP framework which transfers flexibly to both vision-language understanding and generation tasks. BLIP effectively utilizes the noisy web data by bootstrapping the captions, where a captioner generates synthetic captions and a filter removes the noisy ones. We achieve state-of-the-art results on a wide range of vision-language tasks, such as image-text retrieval (+2.7% in average recall@1), image captioning (+2.8% in CIDEr), and VQA (+1.6% in VQA score). BLIP also demonstrates strong generalization ability when directly transferred to video-language tasks in a zero-shot manner. Code, models, and datasets are released at https://github.com/salesforce/BLIP.

Key facts. Cited by 4,310 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 51 of its 59 references that are in the map. First posted on arXiv 28 Jan 2022.

Builds on

51 of 59 references.

Closest ancestors · 2

Same direction, newest first

  1. 2021Scaling Up Vision-Language Pre-training for Image Captioning24 Nov
  2. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervision24 Aug

Other references · 49

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  6. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  7. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  8. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  9. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  10. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  11. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  12. 2015VQA: Visual Question Answeringcited by 2,489
37 more
  1. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  2. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  3. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,609
  4. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  5. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  6. 2019Self-training with Noisy Student improves ImageNet classificationcited by 1,304
  7. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  8. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  9. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  10. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,175
  11. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 966
  12. 2021VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understandingcited by 596
  13. 2019End-to-End Learning of Visual Representations from Uncurated Instructional Videoscited by 506
  14. 2019Unified Vision-Language Pre-Training for Image Captioning and VQAcited by 494
  15. 2021Less is More: ClipBERT for Video-and-Language Learning via Sparse Samplingcited by 484
  16. 2018A Corpus for Reasoning About Natural Language Grounded in Photographscited by 457
  17. 2018nocaps: novel object captioning at scalecited by 444
  18. 2021Unifying Vision-and-Language Tasks via Text Generationcited by 411
  19. 2016Visual Dialogcited by 377
  20. 2020Large-Scale Adversarial Training for Vision-and-Language Representation Learningcited by 327
  21. 2018Bilinear Attention Networkscited by 277
  22. 2020ActBERT: Learning Global-Local Video-Text Representationscited by 267
  23. 2020Just Ask: Learning to Answer Questions from Millions of Narrated Videoscited by 248
  24. 2020UNIMO: Towards Unified-Modal Understanding and Generation via Cross-Modal Contrastive Learningcited by 239
  25. 2019Not Enough Data? Deep Learning to the Rescue!cited by 174
  26. 2020Support-set bottlenecks for video-text representation learningcited by 170
  27. 2020Data Augmentation using Pre-trained Transformer Modelscited by 168
  28. 2020Hierarchical Conditional Relation Networks for Video Question Answeringcited by 163
  29. 2021Align and Prompt: Video-and-Language Pre-training with Entity Promptscited by 149
  30. 2019Heterogeneous Memory Enhanced Multimodal Attention Model for Video Question Answeringcited by 146
  31. 2021VinVL: Revisiting Visual Representations in Vision-Language Modelscited by 109
  32. 2020Training Question Answering Models From Synthetic Datacited by 95
  33. 2019Large-scale Pretraining for Visual Dialog: A Simple State-of-the-Art Baselinecited by 60
  34. 2020VD-BERT: A Unified Vision and Dialog Transformer with BERTcited by 54
  35. 2020Generative Data Augmentation for Commonsense Reasoningcited by 49
  36. 2019Multi-step Reasoning via Recurrent Dual Attention for Visual Dialogcited by 41
  37. 2020e-SNLI-VE: Corrected Visual-Textual Entailment with Natural Language Explanationscited by 30

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 163; 2023: 934; 2024: 1422; 2025: 1165; 2026: 626.

4,310 papers.

Sort:

By area:

  1. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  2. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  3. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  4. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  5. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  6. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  7. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,777
  8. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  9. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  10. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  11. 2022Survey of Hallucination in Natural Language Generationcited by 2,056
  12. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
38 more
  1. 2024HunyuanVideo: A Systematic Framework For Large Video Generative Modelscited by 1,597
  2. 2023IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Modelscited by 1,439
  3. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,385
  4. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
  5. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
  6. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,173
  7. 2022Diffusion Models: A Comprehensive Survey of Methods and Applicationscited by 1,107
  8. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
  9. 2024Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Modelcited by 1,051
  10. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,047
  11. 2023ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generationcited by 1,010
  12. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,000
  13. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 929
  14. 2024Grounded SAM: Assembling Open-World Models for Diverse Visual Taskscited by 914
  15. 2023A Survey on Multimodal Large Language Modelscited by 898
  16. 2023VideoChat: Chat-Centric Video Understandingcited by 851
  17. 2023DreamGaussian: Generative Gaussian Splatting for Efficient 3D Content Creationcited by 837
  18. 2023LRM: Large Reconstruction Model for Single Image to 3Dcited by 834
  19. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 776
  20. 2024Emu3: Next-Token Prediction is All You Needcited by 761
  21. 2024Autoregressive Model Beats Diffusion: Llama for Scalable Image Generationcited by 760
  22. 2023Multimodal Chain-of-Thought Reasoning in Language Modelscited by 740
  23. 2024DoRA: Weight-Decomposed Low-Rank Adaptationcited by 740
  24. 2023Attend-and-Excite: Attention-Based Semantic Guidance for Text-to-Image Diffusion Modelscited by 696
  25. 2023VILA: On Pre-training for Visual Language Modelscited by 686
  26. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decodingcited by 671
  27. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Modelscited by 659
  28. 2022When and why vision-language models behave like bags-of-words, and what to do about it?cited by 603
  29. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  30. 2023Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Modelscited by 591
  31. 2023Structure and Content-Guided Video Synthesis with Diffusion Modelscited by 589
  32. 2024MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?cited by 573
  33. 2024LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Modelscited by 571
  34. 2022MedCLIP: Contrastive Learning from Unpaired Medical Images and Textcited by 564
  35. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 560
  36. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 536
  37. 2023MovieChat: From Dense Token to Sparse Memory for Long Video Understandingcited by 536
  38. 2023MotionGPT: Human Motion as a Foreign Languagecited by 521

50 most cited of 4,310 All 4,310 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Learning by Correction: Efficient Tuning Task for Zero-Shot Generative Vision-Language Reasoning
  2. 2024How Vision-Language Tasks Benefit from Large Pre-trained Models: A Survey
  3. 2024DeepSeek-VL: Towards Real-World Vision-Language Understanding
  4. 2022VLUE: A Multi-Task Benchmark for Evaluating Vision-Language Models
  5. 2025TULIP: Towards Unified Language-Image Pretraining
  6. 2025uCLIP: Parameter-Efficient Multilingual Extension of Vision-Language Models with Unpaired Data
  7. 2022VLP: A Survey on Vision-Language Pre-training
  8. 202620/20 Vision Language Models: A Prescription for Better VLMs through Data Curation Alone