mlsanityData to 7 Oct 2026

Key-point Guided Deformable Image Manipulation Using Diffusion Model

arXiv 2401.08178 ↗First posted 16 Jan 2024cs.CVcited by 0

On the map: Diffusion models → Image generation, text-to-image and editing

Abstract (arXiv)

In this paper, we introduce a Key-point-guided Diffusion probabilistic Model (KDM) that gains precise control over images by manipulating the object's key-point. We propose a two-stage generative model incorporating an optical flow map as an intermediate output. By doing so, a dense pixel-wise understanding of the semantic relation between the image and sparse key point is configured, leading to more realistic image generation. Additionally, the integration of optical flow helps regulate the inter-frame variance of sequential images, demonstrating an authentic sequential image generation. The KDM is evaluated with diverse key-point conditioned image synthesis tasks, including facial image generation, human pose synthesis, and echocardiography video prediction, demonstrating the KDM is proving consistency enhanced and photo-realistic images compared with state-of-the-art models.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. Builds on 38 of its 57 references that are in the map. First posted on arXiv 16 Jan 2024.

Builds on

38 of 57 references.

Closest ancestors · 5

Same direction, newest first

  1. 2023DragDiffusion: Harnessing Diffusion Models for Interactive Point-based Image Editing26 Jun
  2. 2022Paint by Example: Exemplar-based Image Editing with Diffusion Models23 Nov
  3. 2022DiffEdit: Diffusion-based semantic image editing with mask guidance20 Oct
  4. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models20 Dec
  5. 2021Palette: Image-to-Image Diffusion Models10 Nov

Other references · 33

  1. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  3. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  5. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  6. 2020Denoising Diffusion Implicit Modelscited by 9,655
  7. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  8. 2022Classifier-Free Diffusion Guidancecited by 5,717
  9. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,409
  10. 2020RAFT: Recurrent All-Pairs Field Transforms for Optical Flowcited by 2,385
  11. 2016Realtime Multi-Person 2D Pose Estimation using Part Affinity Fieldscited by 1,061
  12. 2015FlowNet: Learning Optical Flow with Convolutional Networkscited by 1,054
21 more
  1. 2017PWC-Net: CNNs for Optical Flow Using Pyramid, Warping, and Cost Volumecited by 966
  2. 2016FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networkscited by 780
  3. 2021Pivotal Tuning for Latent-based Editing of Real Imagescited by 427
  4. 2017Pose Guided Person Image Generationcited by 251
  5. 2018Competitive Collaboration: Joint Unsupervised Learning of Depth, Camera Motion, Optical Flow and Motion Segmentationcited by 222
  6. 2023Drag Your GAN: Interactive Point-based Manipulation on the Generative Image Manifoldcited by 218
  7. 2017Deformable GANs for Pose-based Human Image Generationcited by 166
  8. 2018DF-Net: Unsupervised Joint Learning of Depth and Flow using Cross-Task Consistencycited by 148
  9. 2020Learning by Analogy: Reliable Supervision from Transformations for Unsupervised Optical Flow Estimationcited by 120
  10. 2021Image-to-Image Translation: Methods and Applicationscited by 116
  11. 2022Person Image Synthesis via Denoising Diffusion Modelcited by 104
  12. 2017Occlusion Aware Unsupervised Learning of Optical Flowcited by 101
  13. 2018Models Matter, So Does Training: An Empirical Study of CNNs for Optical Flow Estimationcited by 94
  14. 2019DDFlow: Learning Optical Flow with Unlabeled Data Distillationcited by 63
  15. 2019Stabilizing Generative Adversarial Networks: A Surveycited by 57
  16. 2021DiffuseMorph: Unsupervised Deformable Image Registration Using Diffusion Modelcited by 57
  17. 20223DDesigner: Towards Photorealistic 3D Object Generation and Editing with Text-guided Diffusion Modelscited by 57
  18. 2019Cycle In Cycle Generative Adversarial Networks for Keypoint-Guided Image Generationcited by 34
  19. 2022User-Controllable Latent Transformer for StyleGAN Image Layout Editingcited by 28
  20. 2018Every Smile is Unique: Landmark-Guided Diverse Smile Generationcited by 23
  21. 2017GP-GAN: Gender Preserving GAN for Synthesizing Faces from Landmarkscited by 8

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Deformable 3D Shape Diffusion Model
  2. 2025UniMC: Taming Diffusion Transformer for Unified Keypoint-Guided Multi-Class Image Generation
  3. 2023Image Inpainting via Tractable Steering of Diffusion Models
  4. 2023Motion-Conditioned Diffusion Model for Controllable Video Synthesis
  5. 2024Motion Guidance: Diffusion-Based Image Editing with Differentiable Motion Estimators
  6. 2026StructDiff: A Structure-Preserving and Spatially Controllable Diffusion Model for Single-Image Generation
  7. 2023Guided Image Synthesis via Initial Image Editing in Diffusion Model
  8. 2021More Control for Free! Image Synthesis with Semantic Diffusion Guidance