mlsanityData to 14 Oct 2026

BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models

arXiv 2301.12597 ↗First posted 30 Jan 2023cs.CVcited by 6,137

On the map: Synthetic data and self-improvement for LLMs → Self-improving MLLMs and synthetic data

Abstract (arXiv)

The cost of vision-and-language pre-training has become increasingly prohibitive due to end-to-end training of large-scale models. This paper proposes BLIP-2, a generic and efficient pre-training strategy that bootstraps vision-language pre-training from off-the-shelf frozen pre-trained image encoders and frozen large language models. BLIP-2 bridges the modality gap with a lightweight Querying Transformer, which is pre-trained in two stages. The first stage bootstraps vision-language representation learning from a frozen image encoder. The second stage bootstraps vision-to-language generative learning from a frozen language model. BLIP-2 achieves state-of-the-art performance on various vision-language tasks, despite having significantly fewer trainable parameters than existing methods. For example, our model outperforms Flamingo80B by 8.7% on zero-shot VQAv2 with 54x fewer trainable parameters. We also demonstrate the model's emerging capabilities of zero-shot image-to-text generation that can follow natural language instructions.

Key facts. Cited by 6,137 papers in the map (data through 14 Oct 2026). Placed in Self-improving MLLMs and synthetic data, one of 13 directions in Synthetic data and self-improvement for LLMs. Builds on 39 of its 48 references that are in the map. First posted on arXiv 30 Jan 2023.

Builds on

39 of 48 references.

References · 39

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  5. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  6. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  7. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  8. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  9. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  10. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  11. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
  12. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
27 more
  1. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,619
  2. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,555
  3. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,553
  4. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,362
  5. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,274
  6. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,216
  7. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,214
  8. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,121
  9. 2021Florence: A New Foundation Model for Computer Visioncited by 804
  10. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 766
  11. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 745
  12. 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 691
  13. 2019Unified Language Model Pre-training for Natural Language Understanding and Generationcited by 670
  14. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 661
  15. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 647
  16. 2021FILIP: Fine-grained Interactive Language-Image Pre-Trainingcited by 627
  17. 2021LiT: Zero-Shot Transfer with Locked-image text Tuningcited by 577
  18. 2022Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Taskscited by 511
  19. 2018nocaps: novel object captioning at scalecited by 444
  20. 2021VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Expertscited by 425
  21. 2021Unifying Vision-and-Language Tasks via Text Generationcited by 412
  22. 2021VisualGPT: Data-efficient Adaptation of Pretrained Language Models for Image Captioningcited by 191
  23. 2021VinVL: Revisiting Visual Representations in Vision-Language Modelscited by 110
  24. 2021A Good Prompt Is Worth Millions of Parameters: Low-resource Prompt-based Learning for Vision-Language Modelscited by 101
  25. 2022Plug-and-Play VQA: Zero-shot VQA by Conjoining Large Pretrained Models with Zero Trainingcited by 88
  26. 2022Enabling Multimodal Generation on CLIP via Vision-Language Knowledge Distillationcited by 70
  27. 2022MAPL: Parameter-Efficient Adaptation of Unimodal Pre-Trained Models for Vision-Language Few-Shot Promptingcited by 41

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 948; 2024: 1913; 2025: 1961; 2026: 1315.

6,137 papers.

Sort:

By area:

  1. 2023Visual Instruction Tuningcited by 7,869
  2. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  3. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  4. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 3,080
  5. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,887
  6. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  7. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  8. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
  9. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,224
  10. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,212
  11. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,141
  12. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
38 more
  1. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,729
  2. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,652
  3. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,471
  4. 2023IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Modelscited by 1,464
  5. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,439
  6. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,413
  7. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,402
  8. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,239
  9. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,190
  10. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,109
  11. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,089
  12. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,062
  13. 2024Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Modelcited by 1,056
  14. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,050
  15. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,040
  16. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,005
  17. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 940
  18. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 914
  19. 2023A Survey on Multimodal Large Language Modelscited by 909
  20. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 906
  21. 2023VideoChat: Chat-Centric Video Understandingcited by 857
  22. 2023LRM: Large Reconstruction Model for Single Image to 3Dcited by 849
  23. 2023DreamGaussian: Generative Gaussian Splatting for Efficient 3D Content Creationcited by 843
  24. 2023EVA-CLIP: Improved Training Techniques for CLIP at Scalecited by 798
  25. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 781
  26. 2024Emu3: Next-Token Prediction is All You Needcited by 781
  27. 2023LISA: Reasoning Segmentation via Large Language Modelcited by 776
  28. 2024VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMscited by 771
  29. 2023VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Modelscited by 750
  30. 2025Qwen2.5-Omni Technical Reportcited by 736
  31. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMscited by 735
  32. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 712
  33. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 708
  34. 2023CogAgent: A Visual Language Model for GUI Agentscited by 704
  35. 2023VILA: On Pre-training for Visual Language Modelscited by 692
  36. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)cited by 681
  37. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Modelscited by 668
  38. 2024LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understandingcited by 665

50 most cited of 6,137 All 6,137 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026LeVLJEPA: End-to-End Vision-Language Pretraining Without Negatives
  2. 2023Expedited Training of Visual Conditioned Language Generation via Redundancy Reduction
  3. 2023Stop Pre-Training: Adapt Visual-Language Models to Unseen Languages
  4. 2026Let ViT Speak: Generative Language-Image Pre-training
  5. 2024RWKV-CLIP: A Robust Vision-Language Representation Learner
  6. 2023Effective End-to-End Vision Language Pretraining with Semantic Visual Loss
  7. 2022Generalizing Multimodal Pre-training into Multilingual via Language Acquisition
  8. 2025uCLIP: Parameter-Efficient Multilingual Extension of Vision-Language Models with Unpaired Data