mlsanityData to 7 Oct 2026

DiffMorph: Text-less Image Morphing with Diffusion Models

arXiv 2401.00739 ↗First posted 1 Jan 2024cs.CV cs.AIcited by 0

On the map: Diffusion models → Image generation, text-to-image and editing

Abstract (arXiv)

Text-conditioned image generation models are a prevalent use of AI image synthesis, yet intuitively controlling output guided by an artist remains challenging. Current methods require multiple images and textual prompts for each object to specify them as concepts to generate a single customized image. On the other hand, our work, verb|DiffMorph|, introduces a novel approach that synthesizes images that mix concepts without the use of textual prompts. Our work integrates a sketch-to-image module to incorporate user sketches as input. verb|DiffMorph| takes an initial image with conditioning artist-drawn sketches to generate a morphed image. We employ a pre-trained text-to-image diffusion model and fine-tune it to reconstruct each image faithfully. We seamlessly merge images and concepts from sketches into a cohesive composition. The image generation capability of our work is demonstrated through our results and a comparison of these with prompt-based image generation.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. Builds on 33 of its 43 references that are in the map. First posted on arXiv 1 Jan 2024.

Builds on

33 of 43 references.

Closest ancestors · 10

Same direction, newest first

  1. 2023Adding Conditional Control to Text-to-Image Diffusion Models10 Feb
  2. 2022Multi-Concept Customization of Text-to-Image Diffusion8 Dec
  3. 2022InstructPix2Pix: Learning to Follow Image Editing Instructions17 Nov
  4. 2022eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoisers2 Nov
  5. 2022Imagic: Text-Based Real Image Editing with Diffusion Models17 Oct
  6. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation25 Aug
  7. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding23 May
  8. 2022Hierarchical Text-Conditional Image Generation with CLIP Latents13 Apr
  9. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models20 Dec
  10. 2021Palette: Image-to-Image Diffusion Models10 Nov

Other references · 23

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  5. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  6. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 5,986
  7. 2016Image-to-Image Translation with Conditional Adversarial Networkscited by 4,510
  8. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
  9. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  10. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  11. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  12. 20163D U-Net: Learning Dense Volumetric Segmentation from Sparse Annotationcited by 1,471
11 more
  1. 2019Semantic Image Synthesis with Spatially-Adaptive Normalizationcited by 1,295
  2. 2021Variational Diffusion Modelscited by 1,145
  3. 2016ConceptNet 5.5: An Open Multilingual Graph of General Knowledgecited by 1,065
  4. 2017StarGAN: Unified Generative Adversarial Networks for Multi-Domain Image-to-Image Translationcited by 1,036
  5. 2021StyleCLIP: Text-Driven Manipulation of StyleGAN Imagerycited by 990
  6. 2020Pre-Trained Image Processing Transformercited by 891
  7. 2020Encoding in Style: a StyleGAN Encoder for Image-to-Image Translationcited by 727
  8. 2023HyperDreamBooth: HyperNetworks for Fast Personalization of Text-to-Image Modelscited by 224
  9. 2023Key-Locked Rank One Editing for Text-to-Image Personalizationcited by 194
  10. 2022"This is my unicorn, Fluffy": Personalizing frozen vision-language representationscited by 95
  11. 2019Photo-Sketching: Inferring Contour Drawings from Imagescited by 67

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2023DiffMorpher: Unleashing the Capability of Diffusion Models for Image Morphing
  2. 2023DiffSketching: Sketch Control Image Synthesis with Diffusion Models
  3. 2023Text-to-image Diffusion Models in Generative AI: A Survey
  4. 2025DiffBrush:Just Painting the Art by Your Hands
  5. 2022DiffEdit: Diffusion-based semantic image editing with mask guidance
  6. 2023BoxDiff: Text-to-Image Synthesis with Training-Free Box-Constrained Diffusion
  7. 2023Diffusion Brush: A Latent Diffusion Model-based Editing Tool for AI-generated Images
  8. 2022Text-Guided Synthesis of Artistic Images with Retrieval-Augmented Diffusion Models