mlsanityData to 7 Oct 2026

xGen-MM (BLIP-3): A Family of Open Large Multimodal Models

arXiv 2408.08872 ↗First posted 16 Aug 2024cs.CV cs.AI cs.CLcited by 116

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

This paper introduces BLIP-3, an open framework for developing Large Multimodal Models (LMMs). The framework comprises meticulously curated datasets, a training recipe, model architectures, and a resulting suite of LMMs. We release 4B and 14B models, including both the pre-trained base model and the instruction fine-tuned ones. Our models undergo rigorous evaluation across a range of tasks, including both single and multi-image benchmarks. Our models demonstrate competitive performance among open-source LMMs with similar model sizes. Our resulting LMMs demonstrate competitive performance among open-source LMMs with similar model sizes, with the ability to comprehend interleaved image-text inputs. Our training code, models, and all datasets used in this work, including the three largescale datasets we create and the preprocessed ones, will be open-sourced to better support the research community.

Key facts. Cited by 116 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 67 of its 87 references that are in the map. First posted on arXiv 16 Aug 2024.

Builds on

67 of 87 references.

Closest ancestors · 22

Same direction, newest first

  1. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phone3 Aug
  2. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs24 Jun
  3. 2024MMDU: A Multi-Turn Multi-Image Dialog Understanding Benchmark and Instruction-Tuning Dataset for LVLMs17 Jun
  4. 2024MINT-1T: Scaling Open-Source Multimodal Data by 10x: A Multimodal Dataset with One Trillion Tokens17 Jun
  5. 2024MuirBench: A Comprehensive Benchmark for Robust Multi-image Understanding13 Jun
  6. 2024MANTIS: Interleaved Multi-Image Instruction Tuning2 May
  7. 2024List Items One by One: A New Data Source and Learning Paradigm for Multimodal LLMs25 Apr
  8. 2024BLINK: Multimodal Large Language Models Can See but Not Perceive18 Apr
  9. 2024InternLM-XComposer2-4KHD: A Pioneering Large Vision-Language Model Handling Resolutions from 336 Pixels to 4K HD9 Apr
  10. 2024MM1: Methods, Analysis & Insights from Multimodal LLM Pre-training14 Mar
  11. 2024SPHINX-X: Scaling Data and Parameters for a Family of Multi-modal Large Language Models8 Feb
  12. 2023Generative Multimodal Models are In-Context Learners20 Dec
10 more
  1. 2023VILA: On Pre-training for Visual Language Models12 Dec
  2. 2023Honeybee: Locality-enhanced Projector for Multimodal LLM11 Dec
  3. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captions21 Nov
  4. 2023mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaboration7 Nov
  5. 2023Improved Baselines with Visual Instruction Tuning5 Oct
  6. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension30 Jul
  7. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul
  8. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models23 Jun
  9. 2023OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documents21 Jun
  10. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning11 May

Other references · 45

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  3. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  4. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  5. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  6. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  7. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  8. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  9. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  10. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  11. 2019Towards VQA Models That Can Readcited by 1,696
  12. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
33 more
  1. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,582
  2. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,449
  3. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,445
  4. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,288
  5. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  6. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
  7. 2016Modeling Context in Referring Expressionscited by 1,054
  8. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  9. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 893
  10. 2016A Diagram Is Worth A Dozen Imagescited by 793
  11. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 702
  12. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 694
  13. 2023DataComp: In search of the next generation of multimodal datasetscited by 635
  14. 2023HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Modelscited by 572
  15. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 560
  16. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  17. 2023OCRBench: On the Hidden Mystery of OCR in Large Multimodal Modelscited by 466
  18. 2024VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Modelscited by 443
  19. 2018DVQA: Understanding Data Visualizations via Question Answeringcited by 338
  20. 2024What matters when building vision-language models?cited by 321
  21. 2023Recognize Anything: A Strong Image Tagging Modelcited by 305
  22. 2023MIMIC-IT: Multi-Modal In-Context Instruction Tuningcited by 264
  23. 2023MultiModal-GPT: A Vision and Language Model for Dialogue with Humanscited by 255
  24. 2023PaLI-X: On Scaling up a Multilingual Vision and Language Modelcited by 254
  25. 2023Q-Bench: A Benchmark for General-Purpose Foundation Models on Low-level Visioncited by 247
  26. 2023Data Filtering Networkscited by 247
  27. 2023SVIT: Scaling up Visual Instruction Tuningcited by 161
  28. 2024Orca-Math: Unlocking the potential of SLMs in Grade School Mathcited by 151
  29. 2023To See is to Believe: Prompting GPT-4V for Better Visual Instruction Tuningcited by 143
  30. 2023M3IT: A Large-Scale Dataset towards Multi-Modal Multilingual Instruction Tuningcited by 126
  31. 2024Ferret-v2: An Improved Baseline for Referring and Grounding with Large Language Modelscited by 97
  32. 2023X-InstructBLIP: A Framework for aligning X-Modal instruction-aware representations to LLMs and Emergent Cross-modal Reasoningcited by 78
  33. 2022OCR-IDL: OCR Annotations for Industry Document Library Datasetcited by 35

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 34; 2025: 66; 2026: 16.

116 papers.

Sort:

By area:

  1. 2025Emerging Properties in Unified Multimodal Pretrainingcited by 855
  2. 2025VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understandingcited by 502
  3. 2024Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Modelscited by 321
  4. 2024Building and better understanding vision-language models: insights and future directionscited by 169
  5. 2024Identity-Preserving Text-to-Video Generation by Frequency Decompositioncited by 155
  6. 2025GR-3 Technical Reportcited by 110
  7. 2024FastVLM: Efficient Vision Encoding for Vision Language Modelscited by 76
  8. 2024MM1.5: Methods, Analysis & Insights from Multimodal LLM Fine-tuningcited by 75
  9. 2025Interleave-VLA: Enhancing Robot Manipulation with Interleaved Image-Text Instructionscited by 50
  10. 2024AV-Odyssey Bench: Can Your Multimodal LLMs Really Understand Audio-Visual Information?cited by 46
  11. 2025Pure Vision Language Action (VLA) Models: A Comprehensive Surveycited by 46
  12. 2024ChartMoE: Mixture of Diversely Aligned Expert Connector for Chart Understandingcited by 44
38 more
  1. 2025FineCIR: Explicit Parsing of Fine-Grained Modification Semantics for Composed Image Retrievalcited by 41
  2. 2024VisOnlyQA: Large Vision Language Models Still Struggle with Visual Perception of Geometric Informationcited by 39
  3. 2024xGen-MM-Vid (BLIP-3-Video): You Only Need 32 Tokens to Represent a Video Even in VLMscited by 34
  4. 2025VisualToolAgent (VisTA): A Reinforcement Learning Framework for Visual Tool Selectioncited by 34
  5. 2025FireEdit: Fine-grained Instruction-based Image Editing via Region-aware Vision Language Modelcited by 31
  6. 2024ChatRex: Taming Multimodal LLM for Joint Perception and Understandingcited by 30
  7. 2025SlowFast-LLaVA-1.5: A Family of Token-Efficient Video Large Language Models for Long-Form Video Understandingcited by 30
  8. 2025AToken: A Unified Tokenizer for Visioncited by 30
  9. 2024ICT: Image-Object Cross-Level Trusted Intervention for Mitigating Object Hallucination in Large Vision-Language Modelscited by 28
  10. 2024EventGPT: Event Stream Understanding with Multimodal Large Language Modelscited by 27
  11. 2025LLaVA-Scissor: Token Compression with Semantic Connected Components for Video LLMscited by 25
  12. 2024OBI-Bench: Can LMMs Aid in Study of Ancient Script on Oracle Bones?cited by 22
  13. 2024ProVision: Programmatically Scaling Vision-centric Instruction Data for Multimodal Language Modelscited by 21
  14. 2025DynamicEarth: How Far are We from Open-Vocabulary Change Detection?cited by 21
  15. 2025Valley2: Exploring Multimodal Models with Scalable Vision-Language Designcited by 20
  16. 2024LinVT: Empower Your Image-level Large Language Model to Understand Videoscited by 19
  17. 2025UNIDOC-BENCH: A Unified Benchmark for Document-Centric Multimodal RAGcited by 18
  18. 2025A Survey of Foundation Model-Powered Recommender Systems: From Feature-Based, Generative to Agentic Paradigmscited by 17
  19. 2025MANZANO: A Simple and Scalable Unified Multimodal Model with a Hybrid Vision Tokenizercited by 15
  20. 2024Toward a Holistic Evaluation of Robustness in CLIP Modelscited by 13
  21. 2024Is this Generated Person Existed in Real-world? Fine-grained Detecting and Calibrating Abnormal Human-bodycited by 12
  22. 2025NAVER: A Neuro-Symbolic Compositional Automaton for Visual Grounding with Explicit Logic Reasoningcited by 12
  23. 2025Aligning Multimodal LLM with Human Preference: A Surveycited by 12
  24. 2024Towards Unifying Understanding and Generation in the Era of Vision Foundation Models: A Survey from the Autoregression Perspectivecited by 10
  25. 2025Chitrarth: Bridging Vision and Language for a Billion Peoplecited by 10
  26. 2025Judging the Judges: Can Large Vision-Language Models Fairly Evaluate Chart Comprehension and Reasoning?cited by 10
  27. 2025MSTS: A Multimodal Safety Test Suite for Vision-Language Modelscited by 9
  28. 2025Open-Qwen2VL: Compute-Efficient Pre-Training of Fully-Open Multimodal LLMs on Academic Resourcescited by 9
  29. 2025Towards Understanding Visual Grounding in Visual Language Modelscited by 9
  30. 2024One missing piece in Vision and Language: A Survey on Comics Understandingcited by 8
  31. 2024BLIP3-KALE: Knowledge Augmented Large-Scale Dense Captionscited by 8
  32. 2025LLaVA-SP: Enhancing Visual Representation with Visual Spatial Tokens for MLLMscited by 8
  33. 2025SyncLoop: A Multimodal Dual-Loop Framework for Self-Improving Mathematical Reasoningcited by 8
  34. 2025OmniDPO: A Preference Optimization Framework to Address Omni-Modal Hallucinationcited by 8
  35. 2025Seeing is Understanding: Unlocking Causal Attention into Modality-Mutual Attention for Multimodal LLMscited by 7
  36. 2025Will Pre-Training Ever End? A First Step Toward Next-Generation Foundation MLLMs via Self-Improving Systematic Cognitioncited by 7
  37. 2025ActionArt: Advancing Multimodal Large Models for Fine-Grained Human-Centric Video Understandingcited by 7
  38. 2025IndicVisionBench: Benchmarking Cultural and Multilingual Understanding in VLMscited by 7

50 most cited of 116 All 116 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025BLIP3-o: A Family of Fully Open Unified Multimodal Models-Architecture, Training and Dataset
  2. 2023Large Multimodal Models: Notes on CVPR 2023 Tutorial
  3. 2023An Empirical Study of Scaling Instruct-Tuned Large Multimodal Models
  4. 2023BenchLMM: Benchmarking Cross-style Visual Capability of Large Multimodal Models
  5. 2024LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Models
  6. 2025LMM4LMM: Benchmarking and Evaluating Large-multimodal Image Generation with LMMs
  7. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling
  8. 2024TinyLLaVA: A Framework of Small-scale Large Multimodal Models