mlsanityData to 14 Oct 2026

Diffusion Self-Distillation for Zero-Shot Customized Image Generation

arXiv 2411.18616 ↗First posted 27 Nov 2024cs.CV cs.AI cs.GRcited by 47

On the map: Diffusion models → Image generation, text-to-image and editing

Abstract (arXiv)

Text-to-image diffusion models produce impressive results but are frustrating tools for artists who desire fine-grained control. For example, a common use case is to create images of a specific instance in novel contexts, i.e., "identity-preserving generation". This setting, along with many other tasks (e.g., relighting), is a natural fit for image+text-conditional generative models. However, there is insufficient high-quality paired data to train such a model directly. We propose Diffusion Self-Distillation, a method for using a pre-trained text-to-image model to generate its own dataset for text-conditioned image-to-image tasks. We first leverage a text-to-image diffusion model's in-context generation ability to create grids of images and curate a large paired dataset with the help of a Visual-Language Model. We then fine-tune the text-to-image model into a text+image-to-image model using the curated paired dataset. We demonstrate that Diffusion Self-Distillation outperforms existing zero-shot methods and is competitive with per-instance tuning techniques on a wide range of identity-preservation generation tasks, without requiring test-time optimization.

Key facts. Cited by 47 papers in the map (data through 14 Oct 2026). Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. Builds on 48 of its 51 references that are in the map. First posted on arXiv 27 Nov 2024.

Builds on

48 of 51 references.

Closest ancestors · 18

Same direction, newest first

  1. 2024Group Diffusion Transformers are Unsupervised Multitask Learners19 Oct
  2. 2024IPAdapter-Instruct: Resolving Ambiguity in Image-based Conditioning using Instruct Prompts6 Aug
  3. 2024JeDi: Joint-Image Diffusion Models for Finetuning-Free Personalized Text-to-Image Generation8 Jul
  4. 2024ORACLE: Leveraging Mutual Information for Consistent Character Generation with LoRAs in Diffusion Models4 Jun
  5. 2024ControlNet++: Improving Conditional Controls with Efficient Consistency Feedback11 Apr
  6. 2024LCM-Lookahead for Encoder-based Text-to-Image Personalization4 Apr
  7. 2024BootPIG: Bootstrapping Zero-shot Personalized Image Generation Capabilities in Pretrained Diffusion Models25 Jan
  8. 2024InstantID: Zero-shot Identity-Preserving Generation in Seconds15 Jan
  9. 2023IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Models13 Aug
  10. 2023Subject-Diffusion:Open Domain Personalized Text-to-Image Generation without Test-time Fine-tuning21 Jul
  11. 2023Uni-ControlNet: All-in-One Control to Text-to-Image Diffusion Models25 May
  12. 2023T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Models16 Feb
6 more
  1. 2023Adding Conditional Control to Text-to-Image Diffusion Models10 Feb
  2. 2022InstructPix2Pix: Learning to Follow Image Editing Instructions17 Nov
  3. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation25 Aug
  4. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding23 May
  5. 2022Hierarchical Text-Conditional Image Generation with CLIP Latents13 Apr
  6. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models20 Dec

Other references · 30

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  5. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  6. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,181
  7. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  8. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,581
  9. 2024CogVideoX: Text-to-Video Diffusion Models with An Expert Transformercited by 2,261
  10. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,245
  11. 2021SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equationscited by 1,937
  12. 2023AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuningcited by 1,433
18 more
  1. 2024Depth Anything: Unleashing the Power of Large-Scale Unlabeled Datacited by 1,360
  2. 2022CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformerscited by 985
  3. 2023MVDream: Multi-view Diffusion for 3D Generationcited by 938
  4. 2024WorldSimBench: Towards Video Generation Models as World Simulatorscited by 881
  5. 2023Animate Anyone: Consistent and Controllable Image-to-Video Synthesis for Character Animationcited by 688
  6. 2023Generative Multimodal Models are In-Context Learnerscited by 446
  7. 2023BLIP-Diffusion: Pre-trained Subject Representation for Controllable Text-to-Image Generation and Editingcited by 442
  8. 2022Latent Video Diffusion Models for High-Fidelity Long Video Generationcited by 388
  9. 2023Instant3D: Fast Text-to-3D with Sparse-View Generation and Large Reconstruction Modelcited by 378
  10. 2024OminiControl: Minimal and Universal Control for Diffusion Transformercited by 340
  11. 2023DMV3D: Denoising Multi-View Diffusion using 3D Large Reconstruction Modelcited by 214
  12. 2023Subject-driven Text-to-Image Generation via Apprenticeship Learningcited by 212
  13. 2024DreamBench++: A Human-Aligned Benchmark for Personalized Image Generationcited by 131
  14. 2023State of the Art on Diffusion Models for Visual Computingcited by 127
  15. 2024Collaborative Video Diffusion: Consistent Multi-video Generation with Camera Controlcited by 117
  16. 2024Human4DiT: 360-degree Human Video Generation with 4D Diffusion Transformercited by 61
  17. 2022DiffDreamer: Towards Consistent Unsupervised Single-view Scene Extrapolation with Conditional Diffusion Modelscited by 51
  18. 2023Generative Rendering: Controllable 4D-Guided Video Generation with 2D Diffusion Modelscited by 30

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 28; 2026: 18.

47 papers.

Sort:

By area:

  1. 2025Less-to-More Generalization: Unlocking More Controllability by In-Context Generationcited by 154
  2. 2025Mixture of Contexts for Long Video Generationcited by 95
  3. 2025OminiControl2: Efficient Conditioning for Diffusion Transformerscited by 57
  4. 2025CreatiDesign: A Unified Multi-Conditional Diffusion Transformer for Creative Graphic Designcited by 36
  5. 2025Personalized Image Generation with Deep Generative Models: A Decade Surveycited by 20
  6. 2025DreamPoster: A Unified Framework for Image-Conditioned Generative Poster Designcited by 16
  7. 2025Mod-Adapter: Tuning-Free and Versatile Multi-concept Personalization via Modulation Adaptercited by 13
  8. 2025Learning an Image Editing Model without Image Editing Pairscited by 13
  9. 2026Lance: Unified Multimodal Modeling by Multi-Task Synergycited by 12
  10. 2025MUSAR: Exploring Multi-Subject Customization from Single-Subject Dataset via Attention Routingcited by 11
  11. 2025Conceptrol: Concept Control of Zero-shot Personalized Image Generationcited by 8
  12. 2025Learning Straight Flows: Variational Flow Matching for Efficient Generationcited by 8
35 more
  1. 2025Zero-Shot Dynamic Concept Personalization with Grid-Based LoRAcited by 7
  2. 2025Mind-the-Glitch: Visual Correspondence for Detecting Inconsistencies in Subject-Driven Generationcited by 6
  3. 2025IdentityStory: Taming Your Identity-Preserving Generator for Human-Centric Story Generationcited by 6
  4. 2024Grid: Omni Visual Generationcited by 5
  5. 2026Spanning the Visual Analogy Space with a Weight Basis of LoRAscited by 5
  6. 2026AnimeAgent: Is the Multi-Agent via Image-to-Video models a Good Disney Storytelling Artist?cited by 5
  7. 2026HiFi-Inpaint: Towards High-Fidelity Reference-Based Inpainting for Generating Detail-Preserving Human-Product Imagescited by 5
  8. 2025Beyond the Pixels: VLM-based Evaluation of Identity Preservation in Reference-Guided Synthesiscited by 4
  9. 2025MagicQuillV2: Precise and Interactive Image Editing with Layered Visual Cuescited by 4
  10. 2026TP-Blend: Textual-Prompt Attention Pairing for Precise Object-Style Blending in Diffusion Modelscited by 4
  11. 2025Trade-offs in Image Generation: How Do Different Dimensions Interact?cited by 3
  12. 2025In-Context Sync-LoRA for Portrait Video Editingcited by 3
  13. 2025A Unit Enhancement and Guidance Framework for Audio-Driven Avatar Video Generationcited by 2
  14. 2025Composing Concepts from Images and Videos via Concept-prompt Bindingcited by 2
  15. 2026RefAny3D: 3D Asset-Referenced Diffusion Models for Image Generationcited by 2
  16. 2026Transition Flow Matchingcited by 2
  17. 2026Drift Flow Matchingcited by 2
  18. 2025AlignGen: Boosting Personalized Image Generation with Cross-Modality Prior Alignmentcited by 1
  19. 2025Consistent text-to-image generation via scene de-contextualizationcited by 1
  20. 2025Dynamic Image Prompt Adapter for Scalable Zero-shot Personalized Text-to-Image Generationcited by 1
  21. 2026Say Cheese! Detail-Preserving Portrait Collection Generation via Natural Language Editscited by 1
  22. 2026PLACID: Identity-Preserving Multi-Object Compositing via Video Diffusion with Synthetic Trajectoriescited by 1
  23. 2026EmoStory: Emotion-Aware Story Generationcited by 1
  24. 2025Single Image Iterative Subject-driven Generation and Editingcited by 0
  25. 2025Audit & Repair: An Agentic Framework for Consistent Story Visualization in Text-to-Image Diffusion Modelscited by 0
  26. 2025PAT++: a cautionary tale about generative visual augmentation for Object Re-identificationcited by 0
  27. 2025Comparison Reveals Commonality: Customized Image Generation through Contrastive Inversioncited by 0
  28. 2025Insight-A: Attribution-aware for Multimodal Misinformation Detectioncited by 0
  29. 2026PICS: Pairwise Image Compositing with Spatial Interactionscited by 0
  30. 2026Denoise and Align: Diffusion-Driven Foreground Knowledge Prompting for Open-Vocabulary Temporal Action Detectioncited by 0
  31. 2026Squeezing Capacity from Multimodal Large Language Models for Subject-driven Generationcited by 0
  32. 2026Keep The Essentials: Efficient Reference Conditioned Generation via Token Droppingcited by 0
  33. 2026Latent-Identity Tuning in Text-to-Image Personalization Modelscited by 0
  34. 2026Generative Embodied Multiple Behavior Control Systems for Human-like Agentscited by 0
  35. 2026Posterior Flow Matchingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Diffusion Self-Guidance for Controllable Image Generation
  2. 2026Towards Controllable Image Generation through Representation-Conditioned Diffusion Models
  3. 2025Few-Step Distillation for Text-to-Image Generation: A Practical Guide
  4. 2023Filter-Guided Diffusion for Controllable Image Generation
  5. 2023Diffusion Cocktail: Mixing Domain-Specific Diffusion Models for Diversified Image Generations
  6. 2025Few-Step Diffusion via Score identity Distillation
  7. 2023Steered Diffusion: A Generalized Framework for Plug-and-Play Conditional Image Synthesis
  8. 2022Self-Guided Diffusion Models