mlsanityData to 7 Oct 2026

Bootstrapping Vision-Language Learning with Decoupled Language Pre-training

arXiv 2307.07063 ↗First posted 13 Jul 2023cs.CV cs.LGcited by 28

On the map: Synthetic data and self-improvement for LLMs → Self-improving MLLMs and synthetic data

Abstract (arXiv)

We present a novel methodology aimed at optimizing the application of frozen large language models (LLMs) for resource-intensive vision-language (VL) pre-training. The current paradigm uses visual features as prompts to guide language models, with a focus on determining the most relevant visual features for corresponding text. Our approach diverges by concentrating on the language component, specifically identifying the optimal prompts to align with visual features. We introduce the Prompt-Transformer (P-Former), a model that predicts these ideal prompts, which is trained exclusively on linguistic data, bypassing the need for image-text pairings. This strategy subtly bifurcates the end-to-end VL training process into an additional, separate stage. Our experiments reveal that our framework significantly enhances the performance of a robust image-to-text baseline (BLIP-2), and effectively narrows the performance gap between models trained with either 4M or 129M image-text pairs. Importantly, our framework is modality-agnostic and flexible in terms of architectural design, as validated by its successful application in a video learning task using varied base modules. The code will be made available at https://github.com/yiren-jian/BLIText.

Key facts. Cited by 28 papers in the map (data through 7 Oct 2026). Placed in Self-improving MLLMs and synthetic data, one of 13 directions in Synthetic data and self-improvement for LLMs. Builds on 66 of its 75 references that are in the map. First posted on arXiv 13 Jul 2023.

Builds on

66 of 75 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023X-LLM: Bootstrapping Advanced Large Language Models by Treating Multi-Modalities as Foreign Languages7 May
  2. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models30 Jan

Other references · 64

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  6. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  7. 2023Visual Instruction Tuningcited by 7,792
  8. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  9. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  10. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  11. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  12. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
38 more
  1. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  2. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,259
  3. 2015Fast R-CNNcited by 2,930
  4. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  5. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  6. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  7. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
  8. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  9. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  10. 2015Show, Attend and Tell: Neural Image Caption Generation with Visual Attentioncited by 1,421
  11. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  12. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,310
  13. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  14. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
  15. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  16. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  17. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  18. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  19. 2021MDETR -- Modulated Detection for End-to-End Multi-Modal Understandingcited by 811
  20. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
  21. 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 685
  22. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 657
  23. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 645
  24. 2021ClipCap: CLIP Prefix for Image Captioningcited by 557
  25. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 536
  26. 2022Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Taskscited by 510
  27. 2018nocaps: novel object captioning at scalecited by 444
  28. 2021VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Expertscited by 424
  29. 2019VATEX: A Large-Scale, High-Quality Multilingual Dataset for Video-and-Language Researchcited by 412
  30. 2021Unifying Vision-and-Language Tasks via Text Generationcited by 411
  31. 2020Large-Scale Adversarial Training for Vision-and-Language Representation Learningcited by 327
  32. 2021How Much Can CLIP Benefit Vision-and-Language Tasks?cited by 321
  33. 2020Pixel-BERT: Aligning Image Pixels with Text by Deep Multi-Modal Transformerscited by 288
  34. 2021An Empirical Study of Training End-to-End Vision-and-Language Transformerscited by 271
  35. 2021Multi-Grained Vision Language Pre-Training: Aligning Texts with Visual Conceptscited by 255
  36. 2020UNIMO: Towards Unified-Modal Understanding and Generation via Cross-Modal Contrastive Learningcited by 239
  37. 2021Seeing Out of tHe bOx: End-to-End Pre-training for Vision-Language Representation Learningcited by 187
  38. 2022mPLUG: Effective and Efficient Vision-Language Learning by Cross-modal Skip-connectionscited by 185

50 most cited of 64 All 64 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 12; 2025: 13; 2026: 1.

28 papers.

Sort:

By area:

  1. 2024MM-LLMs: Recent Advances in MultiModal Large Language Modelscited by 277
  2. 2024A Survey of Multimodal Large Language Model from A Data-centric Perspectivecited by 73
  3. 2024BRAVE: Broadening the visual encoding of vision-language modelscited by 68
  4. 2024Foundation Models for Video Understanding: A Surveycited by 46
  5. 2024A Comprehensive Survey of Large Language Models and Multimodal Large Language Models in Medicinecited by 45
  6. 2024OpenWebVoyager: Building Multimodal Web Agents via Iterative Real-World Exploration, Feedback and Optimizationcited by 45
  7. 2024MuChoMusic: Evaluating Music Understanding in Multimodal Audio-Language Modelscited by 43
  8. 2025Analyzing and Boosting the Power of Fine-Grained Visual Recognition for Multi-modal Large Language Modelscited by 30
  9. 2024Uncertainty-Aware Evaluation for Vision-Language Modelscited by 28
  10. 2024Accelerating Multimodal Large Language Models by Searching Optimal Vision Token Reductioncited by 28
  11. 2025Temporal Working Memory: Query-Guided Segment Refinement for Enhanced Multimodal Understandingcited by 28
  12. 2023Expedited Training of Visual Conditioned Language Generation via Redundancy Reductioncited by 15
16 more
  1. 2025Pretrained Image-Text Models are Secretly Video Captionerscited by 11
  2. 2023MAFA: Managing False Negatives for Vision-Language Pre-trainingcited by 9
  3. 2025Structured Preference Optimization for Vision-Language Long-Horizon Task Planningcited by 7
  4. 2024OmniFusion Technical Reportcited by 6
  5. 2025MindBridge: Scalable and Cross-Model Knowledge Editing via Memory-Augmented Modalitycited by 6
  6. 2025Overcoming Multi-step Complexity in Multimodal Theory-of-Mind Reasoning: A Scalable Bayesian Plannercited by 5
  7. 2025Music Audio-Visual Question Answering Requires Specialized Multimodal Designscited by 4
  8. 2025Learning Sparsity for Effective and Efficient Music Performance Question Answeringcited by 4
  9. 2025Learning Event Completeness for Weakly Supervised Video Anomaly Detectioncited by 3
  10. 2025MLlm-DR: Towards Explainable Depression Recognition with MultiModal Large Language Modelscited by 3
  11. 2026Weakly Supervised Video Anomaly Detection with Anomaly-Connected Components and Intention Reasoningcited by 3
  12. 2024GluMarker: A Novel Predictive Modeling of Glycemic Control Through Digital Biomarkerscited by 2
  13. 2024ZALM3: Zero-Shot Enhancement of Vision-Language Alignment via In-Context Information in Multi-Turn Multimodal Medical Dialoguecited by 2
  14. 2025Growing Through Experience: Scaling Episodic Grounding in Language Modelscited by 1
  15. 2025Group-CLIP Uncertainty Modeling for Group Re-Identificationcited by 0
  16. 2025ProtoVQA: An Adaptable Prototypical Framework for Explainable Fine-Grained Visual Question Answeringcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Generalizing vision-language models to novel domains: A comprehensive survey
  2. 2023Language Models as Black-Box Optimizers for Vision-Language Models
  3. 2023Stop Pre-Training: Adapt Visual-Language Models to Unseen Languages
  4. 2024Tuning Vision-Language Models with Candidate Labels by Prompt Alignment
  5. 2024Conditional Prototype Rectification Prompt Learning
  6. 2023mBLIP: Efficient Bootstrapping of Multilingual Vision-LLMs
  7. 2022A Survey of Vision-Language Pre-Trained Models
  8. 2025Modular Prompt Learning Improves Vision-Language Models