mlsanityData to 7 Oct 2026

HumanSD: A Native Skeleton-Guided Diffusion Model for Human Image Generation

arXiv 2304.04269 ↗First posted 9 Apr 2023cs.CVcited by 104

On the map: Diffusion models → 3D, point cloud and scene generation

Abstract (arXiv)

Controllable human image generation (HIG) has numerous real-life applications. State-of-the-art solutions, such as ControlNet and T2I-Adapter, introduce an additional learnable branch on top of the frozen pre-trained stable diffusion (SD) model, which can enforce various conditions, including skeleton guidance of HIG. While such a plug-and-play approach is appealing, the inevitable and uncertain conflicts between the original images produced from the frozen SD branch and the given condition incur significant challenges for the learnable branch, which essentially conducts image feature editing for condition enforcement. In this work, we propose a native skeleton-guided diffusion model for controllable HIG called HumanSD. Instead of performing image editing with dual-branch diffusion, we fine-tune the original SD model using a novel heatmap-guided denoising loss. This strategy effectively and efficiently strengthens the given skeleton condition during model training while mitigating the catastrophic forgetting effects. HumanSD is fine-tuned on the assembly of three large-scale human-centric datasets with text-image-pose information, two of which are established in this work. As shown in Figure 1, HumanSD outperforms ControlNet in terms of accurate pose control and image quality, particularly when the given skeleton guidance is sophisticated.

Key facts. Cited by 104 papers in the map (data through 7 Oct 2026). Placed in 3D, point cloud and scene generation, one of 14 directions in Diffusion models. Builds on 47 of its 59 references that are in the map. First posted on arXiv 9 Apr 2023.

Builds on

47 of 59 references.

References · 47

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2020Denoising Diffusion Implicit Modelscited by 9,655
  6. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  7. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  8. 2022Generative Adversarial Networkscited by 5,603
  9. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  10. 2014Going Deeper with Convolutionscited by 4,231
  11. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  12. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
35 more
  1. 2023T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Modelscited by 1,339
  2. 2018Demystifying MMD GANscited by 1,268
  3. 2016Realtime Multi-Person 2D Pose Estimation using Part Affinity Fieldscited by 1,061
  4. 2023GLIGEN: Open-Set Grounded Text-to-Image Generationcited by 774
  5. 2019NTU RGB+D 120: A Large-Scale Benchmark for 3D Human Activity Understandingcited by 530
  6. 2021BABEL: Bodies, Action and Behavior with English Labelscited by 261
  7. 2021GODIVA: Generating Open-DomaIn Videos from nAtural Descriptionscited by 254
  8. 2017Pose Guided Person Image Generationcited by 251
  9. 2019HigherHRNet: Scale-Aware Representation Learning for Bottom-Up Human Pose Estimationcited by 247
  10. 2021Design Guidelines for Prompt Engineering Text-to-Image Generative Modelscited by 216
  11. 2020Controllable Person Image Synthesis with Attribute-Decomposed GANcited by 150
  12. 2023One-Stage 3D Whole-Body Mesh Recovery with Component Aware Transformercited by 128
  13. 2022HuMMan: Multi-Modal 4D Human Dataset for Versatile Sensing and Modelingcited by 115
  14. 2020Deep Image Spatial Transformation for Person Image Generationcited by 100
  15. 2022Synthesizing Coherent Story with Auto-Regressive Latent Diffusion Modelscited by 69
  16. 2022Exploring Dual-task Correlation for Pose Guided Person Image Generationcited by 68
  17. 2023Human-Art: A Versatile Human-Centric Dataset Bridging Natural and Artificial Scenescited by 68
  18. 2021SmoothNet: A Plug-and-Play Network for Refining Human Poses in Videoscited by 64
  19. 2021PISE: Person Image Synthesis and Editing with Decoupled GANcited by 62
  20. 2021Learning Skeletal Graph Neural Networks for Hard 3D Pose Estimationcited by 62
  21. 2022Neural Texture Extraction and Distribution for Controllable Person Image Synthesiscited by 55
  22. 2017Be Your Own Prada: Fashion Synthesis with Structural Coherencecited by 49
  23. 2023Explicit Box Detection Unifies End-to-End Multi-Person Pose Estimationcited by 48
  24. 2022Cross Attention Based Style Distribution for Controllable Person Image Synthesiscited by 45
  25. 2021Learning Semantic Person Image Generation by Region-Adaptive Normalizationcited by 44
  26. 2020HAA500: Human-Centric Atomic Action Dataset with Curated Videoscited by 38
  27. 2019HAKE: Human Activity Knowledge Enginecited by 32
  28. 2022DeciWatch: A Simple Baseline for 10x Efficient 2D and 3D Pose Estimationcited by 29
  29. 2020Towards Fine-grained Human Pose Transfer with Detail Replenishing Networkcited by 21
  30. 2021CT-Net: Complementary Transfering Network for Garment Transfer with Arbitrary Geometric Changescited by 17
  31. 2020MUST-GAN: Multi-level Statistics Transfer for Self-driven Person Image Generationcited by 16
  32. 2022KPE: Keypoint Pose Encoding for Transformer-based Image Generationcited by 12
  33. 2022TIPS: Text-Induced Pose Synthesiscited by 11
  34. 2022HumanDiffusion: a Coarse-to-Fine Alignment Diffusion Framework for Controllable Text-Driven Person Image Generationcited by 11
  35. 2022SCAM! Transferring humans between images with Semantic Cross Attention Modulationcited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 13; 2024: 45; 2025: 32; 2026: 14.

104 papers.

Sort:

By area:

  1. 2024Grounded SAM: Assembling Open-World Models for Diverse Visual Taskscited by 914
  2. 2023PhotoMaker: Customizing Realistic Human Photos via Stacked ID Embeddingcited by 324
  3. 2023Effective Whole-body Pose Estimation with Two-stages Distillationcited by 307
  4. 2024Semantic Image Inversion and Editing using Rectified Stochastic Differential Equationscited by 162
  5. 2024ControlNet++: Improving Conditional Controls with Efficient Consistency Feedbackcited by 153
  6. 2023Direct Inversion: Boosting Diffusion-based Editing with 3 Lines of Codecited by 145
  7. 2023TeCH: Text-guided Reconstruction of Lifelike Clothed Humanscited by 104
  8. 2025Cosmos-Transfer1: Conditional World Generation with Adaptive Multimodal Controlcited by 95
  9. 2023HumanGaussian: Text-Driven 3D Human Generation with Gaussian Splattingcited by 89
  10. 2023DreamWaltz: Make a Scene with Complex 3D Animatable Avatarscited by 82
  11. 2024Controllable Generation with Text-to-Image Diffusion Models: A Surveycited by 82
  12. 2023HyperHuman: Hyper-Realistic Human Generation with Latent Structural Diffusioncited by 68
38 more
  1. 2024ControlVAR: Exploring Controllable Visual Autoregressive Modelingcited by 58
  2. 2025EditVerse: Unifying Image and Video Editing and Generation with In-Context Learningcited by 53
  3. 2025FullDiT: Multi-Task Video Generative Foundation Model with Full Attentioncited by 45
  4. 2024ConsistentID: Portrait Generation with Multimodal Fine-Grained Identity Preservingcited by 42
  5. 2024BrushEdit: All-In-One Image Inpainting and Editingcited by 31
  6. 2024CosmicMan: A Text-to-Image Foundation Model for Humanscited by 30
  7. 2023TOSS:High-quality Text-guided Novel View Synthesis from a Single Imagecited by 26
  8. 2025RelationAdapter: Learning and Transferring Visual Relation with Diffusion Transformerscited by 26
  9. 2024SmartControl: Enhancing ControlNet for Handling Rough Visual Conditionscited by 24
  10. 2024HumanRefiner: Benchmarking Abnormal Human Generation and Refining with Coarse-to-fine Pose-Reversible Guidancecited by 24
  11. 2024Edify Image: High-Quality Image Generation with Pixel Space Laplacian Diffusion Modelscited by 22
  12. 2024Boosting Camera Motion Control for Video Diffusion Transformerscited by 20
  13. 2025Referring to Any Personcited by 20
  14. 2024ClassDiffusion: More Aligned Personalization Tuning with Explicit Class Guidancecited by 18
  15. 2024Towards Effective Usage of Human-Centric Priors in Diffusion Models for Text-based Human Image Generationcited by 17
  16. 2024MoLE: Enhancing Human-centric Text-to-image Diffusion via Mixture of Low-rank Expertscited by 15
  17. 2024VirtualModel: Generating Object-ID-retentive Human-object Interaction Image by Diffusion Model for E-commerce Marketingcited by 14
  18. 2024GRPose: Learning Graph Relations for Human Image Generation with Pose Priorscited by 14
  19. 2025Simulating the Real World: A Unified Survey of Multimodal Generative Modelscited by 14
  20. 2023A Generalist FaceX via Learning Unified Facial Representationcited by 13
  21. 2024Giving a Hand to Diffusion Models: a Two-Stage Approach to Improving Conditional Human Image Generationcited by 13
  22. 2025Video-As-Prompt: Unified Semantic Control for Video Generationcited by 13
  23. 2024Stable-Pose: Leveraging Transformers for Pose-Guided Text-to-Image Generationcited by 12
  24. 2024DreamWaltz-G: Expressive 3D Gaussian Avatars from Skeleton-Guided 2D Diffusioncited by 12
  25. 2024Is this Generated Person Existed in Real-world? Fine-grained Detecting and Calibrating Abnormal Human-bodycited by 12
  26. 2024IDOL: Unified Dual-Modal Latent Diffusion for Human-Centric Joint Video-Depth Generationcited by 11
  27. 2024DiffLoRA: Generating Personalized Low-Rank Adaptation Weights with Diffusioncited by 11
  28. 2025Human-Centric Foundation Models: Perception, Generation and Agentic Modelingcited by 10
  29. 2024FastVideoEdit: Leveraging Consistency Models for Efficient Text-to-Video Editingcited by 9
  30. 2024BeyondScene: Higher-Resolution Human-Centric Scene Generation With Pretrained Diffusioncited by 9
  31. 2024RAGDiffusion: Faithful Cloth Generation via External Knowledge Assimilationcited by 9
  32. 2025FinePhys: Fine-grained Human Action Generation by Explicitly Incorporating Physical Laws for Effective Skeletal Guidancecited by 9
  33. 2023Generative AIBIM: An automatic and intelligent structural design pipeline integrating BIM and generative AIcited by 8
  34. 2024Diffusion Models are Efficient Data Generators for Human Mesh Recoverycited by 8
  35. 2024SimuScope: Realistic Endoscopic Synthetic Dataset Generation through Surgical Simulation and Diffusion Modelscited by 8
  36. 2023ViscoNet: Bridging and Harmonizing Visual and Textual Conditioning for ControlNetcited by 7
  37. 2024Evaluating Text-to-Image Generative Models: An Empirical Study on Human Image Synthesiscited by 7
  38. 2024Follow-Your-MultiPose: Tuning-Free Multi-Character Text-to-Video Generation via Pose Guidancecited by 7

50 most cited of 104 All 104 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026One-Shot Novel View and Pose Human Image Synthesis via 3D Prior Guided Diffusion Model
  2. 2025Jointly Conditioned Diffusion Model for Multi-View Pose-Guided Person Image Synthesis
  3. 2023Diffusion-HPC: Synthetic Data Generation for Human Mesh Recovery in Challenging Domains
  4. 2022Person Image Synthesis via Denoising Diffusion Model
  5. 2024A Simple Approach to Unifying Diffusion-based Conditional Generation
  6. 2024Controlling Human Shape and Pose in Text-to-Image Diffusion Models via Domain Adaptation
  7. 2024MacDiff: Unified Skeleton Modeling with Masked Conditional Diffusion
  8. 2026Improving Controllable Generation: Faster Training and Better Performance via x0-Supervision