mlsanityData to 7 Oct 2026

BLIP3-o: A Family of Fully Open Unified Multimodal Models-Architecture, Training and Dataset

arXiv 2505.09568 ↗First posted 14 May 2025cs.CV cs.AIcited by 363

On the map: Diffusion models → Image generation, text-to-image and editing

Abstract (arXiv)

Unifying image understanding and generation has gained growing attention in recent research on multimodal models. Although design choices for image understanding have been extensively studied, the optimal model architecture and training recipe for a unified framework with image generation remain underexplored. Motivated by the strong potential of autoregressive and diffusion models for high-quality generation and scalability, we conduct a comprehensive study of their use in unified multimodal settings, with emphasis on image representations, modeling objectives, and training strategies. Grounded in these investigations, we introduce a novel approach that employs a diffusion transformer to generate semantically rich CLIP image features, in contrast to conventional VAE-based representations. This design yields both higher training efficiency and improved generative quality. Furthermore, we demonstrate that a sequential pretraining strategy for unified models-first training on image understanding and subsequently on image generation-offers practical advantages by preserving image understanding capability while developing strong image generation ability. Finally, we carefully curate a high-quality instruction-tuning dataset BLIP3o-60k for image generation by prompting GPT-4o with a diverse set of captions covering various scenes, objects, human gestures, and more. Building on our innovative model design, training recipe, and datasets, we develop BLIP3-o, a suite of state-of-the-art unified multimodal models. BLIP3-o achieves superior performance across most of the popular benchmarks spanning both image understanding and generation tasks. To facilitate future research, we fully open-source our models, including code, model weights, training scripts, and pretraining and instruction tuning datasets.

Key facts. Cited by 363 papers in the map (data through 7 Oct 2026). Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. First posted on arXiv 14 May 2025.

Builds on

0 of 0 references.

No references here yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 154; 2026: 209.

363 papers.

Sort:

By area:

  1. 2025Qwen-Image Technical Reportcited by 973
  2. 2025Diffusion Transformers with Representation Autoencoderscited by 275
  3. 2025Z-Image: An Efficient Image Generation Foundation Model with Single-Stream Diffusion Transformercited by 247
  4. 2025Show-o2: Improved Native Unified Multimodal Modelscited by 242
  5. 2025OmniGen2: Towards Instruction-Aligned Multimodal Generationcited by 241
  6. 2025Thinking with Images for Multimodal Reasoning: Foundations, Methods, and Future Frontierscited by 193
  7. 2025Machine Mental Imagery: Empower Multimodal Reasoning with Latent Visual Tokenscited by 120
  8. 2025Emu3.5: Native Multimodal Models are World Learnerscited by 119
  9. 2025X-Omni: Reinforcement Learning Makes Discrete Autoregressive Image Generative Models Great Againcited by 105
  10. 2025Echo-4o: Harnessing the Power of GPT-4o Synthetic Images for Improved Image Generationcited by 95
  11. 2025ShareGPT-4o-Image: Aligning Multimodal Models with GPT-4o-Level Image Generationcited by 94
  12. 2025Pref-GRPO: Pairwise Preference Reward-based GRPO for Stable Text-to-Image Reinforcement Learningcited by 88
38 more
  1. 2025F1: A Vision-Language-Action Model Bridging Understanding and Generation to Actionscited by 85
  2. 2025UniVideo: Unified Understanding, Generation, and Editing for Videoscited by 85
  3. 2025Lumina-DiMOO: An Omni Diffusion Large Language Model for Multi-Modal Generation and Understandingcited by 83
  4. 2025OneIG-Bench: Omni-dimensional Nuanced Evaluation for Image Generationcited by 81
  5. 2025PixNerd: Pixel Neural Field Diffusioncited by 75
  6. 2025Ovis-U1 Technical Reportcited by 65
  7. 2025UniWorld-V1: High-Resolution Semantic Encoders for Unified Visual Understanding and Generationcited by 64
  8. 2025NextStep-1: Toward Autoregressive Image Generation with Continuous Tokens at Scalecited by 55
  9. 2025DeCo: Frequency-Decoupled Pixel Diffusion for End-to-End Image Generationcited by 55
  10. 2025EditVerse: Unifying Image and Video Editing and Generation with In-Context Learningcited by 53
  11. 2025Muddit: Liberating Generation Beyond Text-to-Image with a Unified Discrete Diffusion Modelcited by 52
  12. 2025Uni-cot: Towards Unified Chain-of-Thought Reasoning Across Text and Visioncited by 52
  13. 2025OneCAT: Decoder-Only Auto-Regressive Model for Unified Understanding and Generationcited by 50
  14. 2025BLIP3o-NEXT: Next Frontier of Native Image Generationcited by 50
  15. 2026Scaling Text-to-Image Diffusion Transformers with Representation Autoencoderscited by 49
  16. 2025Reconstruction Alignment Improves Unified Multimodal Modelscited by 48
  17. 2025FUDOKI: Discrete Flow-based Unified Understanding and Generation via Kinetic-Optimal Velocitiescited by 46
  18. 2025OpenUni: A Simple Baseline for Unified Multimodal Understanding and Generationcited by 45
  19. 2025UniLiP: Adapting CLIP for Unified Multimodal Understanding, Generation and Editingcited by 43
  20. 2025GenEval 2: Addressing Benchmark Drift in Text-to-Image Evaluationcited by 42
  21. 2025How Far are VLMs from Visual Spatial Intelligence? A Benchmark-Driven Perspectivecited by 41
  22. 2025Interleaving Reasoning for Better Text-to-Image Generationcited by 40
  23. 2025Omni-Video: Democratizing Unified Video Understanding and Generationcited by 39
  24. 2025G2VLM: Geometry Grounded Vision Language Model with Unified 3D Reconstruction and Spatial Reasoningcited by 39
  25. 2025Skywork UniPic 2.0: Building Kontext Model with Online RL for Unified Multimodal Modelcited by 38
  26. 2026VLM4VLA: Revisiting Vision-Language-Models in Vision-Language-Action Modelscited by 38
  27. 2025FLUX-Reason-6M & PRISM-Bench: A Million-Scale Text-to-Image Reasoning Dataset and Comprehensive Benchmarkcited by 37
  28. 2025Jigsaw-R1: A Study of Rule-based Visual Reinforcement Learning with Jigsaw Puzzlescited by 36
  29. 2025TUNA: Taming Unified Visual Representations for Native Unified Multimodal Modelscited by 34
  30. 2025Easier Painting Than Thinking: Can Text-to-Image Models Set the Stage, but Not Direct the Play?cited by 33
  31. 2025RealUnify: Do Unified Models Truly Benefit from Unification? A Comprehensive Benchmarkcited by 33
  32. 2025UniGen: Enhanced Training & Test-Time Strategies for Unified Multimodal Understanding and Generationcited by 32
  33. 2025Transition Models: Rethinking the Generative Learning Objectivecited by 32
  34. 2026Representation Fr'echet Loss for Visual Generationcited by 32
  35. 2025InstructX: Towards Unified Visual Editing with MLLM Guidancecited by 31
  36. 2025MindOmni: Unleashing Reasoning Generation in Vision Language Models with RGPOcited by 30
  37. 2025EO-1: An Open Unified Embodied Foundation Model for General Robot Controlcited by 30
  38. 2026HiDream-O1-Image: A Natively Unified Image Generative Foundation Model with Pixel-level Unified Transformercited by 29

50 most cited of 363 All 363 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Dual Diffusion for Unified Image Generation and Understanding
  2. 2024xGen-MM (BLIP-3): A Family of Open Large Multimodal Models
  3. 2025Unified Multimodal Discrete Diffusion
  4. 2024Show-o: One Single Transformer to Unify Multimodal Understanding and Generation
  5. 2024Diffusion Feedback Helps CLIP See Better
  6. 2024UNIMO-G: Unified Image Generation through Multimodal Conditional Diffusion
  7. 2025MammothModa2: A Unified AR-Diffusion Framework for Multimodal Understanding and Generation
  8. 2023BLIP-Diffusion: Pre-trained Subject Representation for Controllable Text-to-Image Generation and Editing