mlsanityData to 7 Oct 2026

BLIP-Diffusion: Pre-trained Subject Representation for Controllable Text-to-Image Generation and Editing

arXiv 2305.14720 ↗First posted 24 May 2023cs.CV cs.AIcited by 438

Not on the map. Why?

Cited by 176 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Subject-driven text-to-image generation models create novel renditions of an input subject based on text prompts. Existing models suffer from lengthy fine-tuning and difficulties preserving the subject fidelity. To overcome these limitations, we introduce BLIP-Diffusion, a new subject-driven image generation model that supports multimodal control which consumes inputs of subject images and text prompts. Unlike other subject-driven generation models, BLIP-Diffusion introduces a new multimodal encoder which is pre-trained to provide subject representation. We first pre-train the multimodal encoder following BLIP-2 to produce visual representation aligned with the text. Then we design a subject representation learning task which enables a diffusion model to leverage such visual representation and generates new subject renditions. Compared with previous methods such as DreamBooth, our model enables zero-shot subject-driven generation, and efficient fine-tuning for customized subject with up to 20x speedup. We also demonstrate that BLIP-Diffusion can be flexibly combined with existing techniques such as ControlNet and prompt-to-prompt to enable novel subject-driven generation and editing applications. Code and models will be released at https://github.com/salesforce/LAVIS/tree/main/projects/blip-diffusion. Project page at https://dxli94.github.io/BLIP-Diffusion-website/.

Key facts. Cited by 438 papers in the map (data through 7 Oct 2026). Builds on 33 of its 40 references that are in the map. First posted on arXiv 24 May 2023.

Builds on

33 of 40 references.

References · 33

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  6. 2020Denoising Diffusion Implicit Modelscited by 9,655
  7. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  8. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  9. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  10. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  11. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  12. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
21 more
  1. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  2. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  3. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  4. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  5. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,286
  6. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  7. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,591
  8. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,409
  9. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,231
  10. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  11. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  12. 2022Multi-Concept Customization of Text-to-Image Diffusioncited by 1,099
  13. 2022Pseudo Numerical Methods for Diffusion Models on Manifoldscited by 695
  14. 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 685
  15. 2021Image Segmentation Using Text and Image Promptscited by 509
  16. 2023InstantBooth: Personalized Text-to-Image Generation without Test-Time Finetuningcited by 343
  17. 2022EDICT: Exact Diffusion Inversion via Coupled Transformationscited by 256
  18. 2023Subject-driven Text-to-Image Generation via Apprenticeship Learningcited by 210
  19. 2022Re-Imagen: Retrieval-Augmented Text-to-Image Generatorcited by 208
  20. 2023Taming Encoder for Zero Fine-tuning Image Customization with Text-to-Image Diffusion Modelscited by 116
  21. 2020PyMatting: A Python Library for Alpha Mattingcited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 43; 2024: 162; 2025: 166; 2026: 67.

438 papers.

Sort:

By area:

  1. 2024Evaluating Text-to-Visual Generation with Image-to-Text Generationcited by 465
  2. 2023Generative Multimodal Models are In-Context Learnerscited by 446
  3. 2023AnyDoor: Zero-shot Object-level Image Customizationcited by 411
  4. 2024OmniGen: Unified Image Generationcited by 388
  5. 2024OminiControl: Minimal and Universal Control for Diffusion Transformercited by 334
  6. 2023DreamLLM: Synergistic Multimodal Comprehension and Creationcited by 296
  7. 2024UltraEdit: Instruction-based Fine-Grained Image Editing at Scalecited by 266
  8. 2025Transfer between Modalities with MetaQueriescited by 231
  9. 2023Style Aligned Image Generation via Shared Attentioncited by 214
  10. 2023VIEScore: Towards Explainable Metrics for Conditional Image Synthesis Evaluationcited by 196
  11. 2023Subject-Diffusion:Open Domain Personalized Text-to-Image Generation without Test-time Fine-tuningcited by 191
  12. 2023Guiding Instruction-based Image Editing via Multimodal Large Language Modelscited by 182
38 more
  1. 2025Less-to-More Generalization: Unlocking More Controllability by In-Context Generationcited by 153
  2. 2023Alpha-CLIP: A CLIP Model Focusing on Wherever You Wantcited by 150
  3. 2023SSR-Encoder: Encoding Selective Subject Representation for Subject-Driven Generationcited by 142
  4. 2024MS-Diffusion: Multi-subject Zero-shot Image Personalization with Layout Guidancecited by 142
  5. 2024GenArtist: Multimodal LLM as an Agent for Unified Image Generation and Editingcited by 132
  6. 2024DreamBench++: A Human-Aligned Benchmark for Personalized Image Generationcited by 131
  7. 2023VideoBooth: Diffusion-based Video Generation with Image Promptscited by 124
  8. 2024Training-Free Consistent Text-to-Image Generationcited by 123
  9. 2025EasyControl: Adding Efficient and Flexible Control for Diffusion Transformercited by 118
  10. 2023Kosmos-G: Generating Images in Context with Multimodal Large Language Modelscited by 105
  11. 2024IMAGDressing-v1: Customizable Virtual Dressingcited by 100
  12. 2024UniReal: Universal Image Generation and Editing via Learning Real-world Dynamicscited by 100
  13. 2024Video-LaVIT: Unified Video-Language Pre-training with Decoupled Visual-Motional Tokenizationcited by 98
  14. 2024DiffEditor: Boosting Accuracy and Flexibility on Diffusion-based Image Editingcited by 88
  15. 2023Domain-Agnostic Tuning-Encoder for Fast Personalization of Text-To-Image Modelscited by 85
  16. 2024Controllable Generation with Text-to-Image Diffusion Models: A Surveycited by 82
  17. 2025DreamO: A Unified Framework for Image Customizationcited by 82
  18. 2024GenAI Arena: An Open Evaluation Platform for Generative Modelscited by 81
  19. 2023Intelligent Grimm -- Open-ended Visual Storytelling via Latent Diffusion Modelscited by 76
  20. 2023ImagenHub: Standardizing the evaluation of conditional image generation modelscited by 75
  21. 2024Natural Language Understanding and Inference with MLLM in Visual Question Answering: A Surveycited by 74
  22. 2023GPT4Point: A Unified Framework for Point-Language Understanding and Generationcited by 69
  23. 2024DEADiff: An Efficient Stylization Diffusion Model with Disentangled Representationscited by 68
  24. 2025ConceptMaster: Multi-Concept Video Customization on Diffusion Transformer Models Without Test-Time Tuningcited by 67
  25. 2024Instruct-Imagen: Image Generation with Multi-modal Instructioncited by 65
  26. 2025Multi-subject Open-set Personalization in Video Generationcited by 65
  27. 2024Ctrl-Adapter: An Efficient and Versatile Framework for Adapting Diverse Controls to Any Diffusion Modelcited by 64
  28. 2023CoDi-2: In-Context, Interleaved, and Interactive Any-to-Any Generationcited by 63
  29. 2023The Chosen One: Consistent Characters in Text-to-Image Diffusion Modelscited by 62
  30. 2024DreamMatcher: Appearance Matching Self-Attention for Semantically-Consistent Text-to-Image Personalizationcited by 60
  31. 2024MyVLM: Personalizing VLMs for User-Specific Queriescited by 58
  32. 2024ControlVAR: Exploring Controllable Visual Autoregressive Modelingcited by 58
  33. 2024CustomCrafter: Customized Video Generation with Preserving Motion and Concept Composition Abilitiescited by 58
  34. 2024LoRA-Composer: Leveraging Low-Rank Adaptation for Multi-Concept Customization in Training-Free Diffusion Modelscited by 55
  35. 2024JeDi: Joint-Image Diffusion Models for Finetuning-Free Personalized Text-to-Image Generationcited by 52
  36. 2024λ-ECLIPSE: Multi-Concept Personalized Text-to-Image Diffusion Models by Leveraging CLIP Latent Spacecited by 51
  37. 2024MoA: Mixture-of-Attention for Subject-Context Disentanglement in Personalized Image Generationcited by 51
  38. 2024Drag Your Noise: Interactive Point-based Editing via Diffusion Semantic Propagationcited by 50

50 most cited of 438 All 438 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025BLIP3-o: A Family of Fully Open Unified Multimodal Models-Architecture, Training and Dataset
  2. 2023DreamDistribution: Learning Prompt Distribution for Diverse In-distribution Generation
  3. 2025SceneBooth: Diffusion-based Framework for Subject-preserved Text-to-Image Generation
  4. 2025Proxy-Tuning: Tailoring Multimodal Autoregressive Models for Subject-Driven Image Generation
  5. 2024Compositional Text-to-Image Generation with Dense Blob Representations
  6. 2023R&B: Region and Boundary Aware Zero-shot Grounded Text-to-image Generation
  7. 2024Mastering Text-to-Image Diffusion: Recaptioning, Planning, and Generating with Multimodal LLMs
  8. 2023FastComposer: Tuning-Free Multi-Subject Image Generation with Localized Attention