mlsanityData to 7 Oct 2026

HanDiffuser: Text-to-Image Generation With Realistic Hand Appearances

arXiv 2403.01693 ↗First posted 4 Mar 2024cs.CV cs.AIcited by 42

On the map: Diffusion models → Diffusion and flow policies for robotics

Abstract (arXiv)

Text-to-image generative models can generate high-quality humans, but realism is lost when generating hands. Common artifacts include irregular hand poses, shapes, incorrect numbers of fingers, and physically implausible finger orientations. To generate images with realistic hands, we propose a novel diffusion-based architecture called HanDiffuser that achieves realism by injecting hand embeddings in the generative process. HanDiffuser consists of two components: a Text-to-Hand-Params diffusion model to generate SMPL-Body and MANO-Hand parameters from input text prompts, and a Text-Guided Hand-Params-to-Image diffusion model to synthesize images by conditioning on the prompts and hand parameters generated by the previous component. We incorporate multiple aspects of hand representation, including 3D shapes and joint-level finger positions, orientations and articulations, for robust learning and reliable performance during inference. We conduct extensive quantitative and qualitative experiments and perform user studies to demonstrate the efficacy of our method in generating images with high-quality hands.

Key facts. Cited by 42 papers in the map (data through 7 Oct 2026). Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models. Builds on 60 of its 78 references that are in the map. First posted on arXiv 4 Mar 2024.

Builds on

60 of 78 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023HandRefiner: Refining Malformed Hands in Generated Images by Diffusion-based Conditional Inpainting29 Nov

Other references · 59

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  6. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  7. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  8. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  9. 2022Classifier-Free Diffusion Guidancecited by 5,717
  10. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  11. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  12. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
38 more
  1. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  2. 2021SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equationscited by 1,903
  3. 2019AMASS: Archive of Motion Capture as Surface Shapescited by 1,208
  4. 2018OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fieldscited by 1,161
  5. 2021Cascaded Diffusion Models for High Fidelity Image Generationcited by 1,155
  6. 2022Human Motion Diffusion Modelcited by 1,062
  7. 2021Vector Quantized Diffusion Model for Text-to-Image Synthesiscited by 773
  8. 2022MotionDiffuse: Text-Driven Human Motion Generation with Diffusion Modelcited by 768
  9. 2022Pseudo Numerical Methods for Diffusion Models on Manifoldscited by 695
  10. 2017AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networkscited by 664
  11. 2019NTU RGB+D 120: A Large-Scale Benchmark for 3D Human Activity Understandingcited by 530
  12. 2021Action-Conditioned 3D Human Motion Synthesis with Transformer VAEcited by 497
  13. 2022Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priorscited by 487
  14. 2022TEMOS: Generating diverse human motions from textual descriptionscited by 486
  15. 2021Pix2seq: A Language Modeling Framework for Object Detectioncited by 345
  16. 2016The KIT Motion-Language Datasetcited by 334
  17. 2019DM-GAN: Dynamic Memory Generative Adversarial Networks for Text-to-Image Synthesiscited by 308
  18. 2022ARCTIC: A Dataset for Dexterous Bimanual Hand-Object Manipulationcited by 306
  19. 2021BABEL: Bodies, Action and Behavior with English Labelscited by 261
  20. 2021Cross-Modal Contrastive Learning for Text-to-Image Generationcited by 259
  21. 2019Language2Pose: Natural Language Grounded Pose Forecastingcited by 257
  22. 2022MoFusion: A Framework for Denoising-Diffusion-based Motion Synthesiscited by 218
  23. 2022FLAME: Free-form Language-based Motion Synthesis & Editingcited by 217
  24. 2020Monocular, One-stage, Regression of Multiple 3D Peoplecited by 216
  25. 2021Design Guidelines for Prompt Engineering Text-to-Image Generative Modelscited by 216
  26. 2022Embodied Hands: Modeling and Capturing Hands and Bodies Togethercited by 179
  27. 2021Synthesis of Compositional Animations from Textual Descriptionscited by 177
  28. 2022ReCo: Region-Controlled Text-to-Image Generationcited by 177
  29. 2020DF-GAN: A Simple and Effective Baseline for Text-to-Image Synthesiscited by 161
  30. 2020MediaPipe Hands: On-device Real-time Hand Trackingcited by 157
  31. 2021GOAL: Generating 4D Whole-Body Motion for Hand-Object Graspingcited by 142
  32. 2015Generating Images from Captions with Attentioncited by 135
  33. 2022IMos: Intent-Driven Full-Body Motion Synthesis for Human-Object Interactionscited by 121
  34. 2017Text2Action: Generative Adversarial Synthesis from Language to Actioncited by 106
  35. 2019Efficient Learning on Point Clouds with Basis Point Setscited by 101
  36. 2022TOCH: Spatio-Temporal Object-to-Hand Correspondence for Motion Refinementcited by 69
  37. 2021Improving Text-to-Image Synthesis Using Contrastive Learningcited by 62
  38. 2022AvatarCLIP: Zero-Shot Text-Driven Generation and Animation of 3D Avatarscited by 55

50 most cited of 59 All 59 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 15; 2025: 14; 2026: 13.

42 papers.

Sort:

By area:

  1. 2024Understanding Hallucinations in Diffusion Models through Mode Interpolationcited by 87
  2. 2024ManiVideo: Generating Hand-Object Manipulation Video with Dexterous and Generalizable Graspingcited by 18
  3. 2024MoLE: Enhancing Human-centric Text-to-image Diffusion via Mixture of Low-rank Expertscited by 15
  4. 2025X-Dancer: Expressive Music to Human Dance Video Generationcited by 15
  5. 2024Detecting Human Artifacts from Text-to-Image Modelscited by 12
  6. 2024Hand1000: Generating Realistic Hands from Text with Only 1,000 Imagescited by 11
  7. 2025Mitigating Hallucinations in Diffusion Models through Adaptive Attention Modulationcited by 11
  8. 2024AttentionHand: Text-driven Controllable Hand Image Generation for 3D Hand Reconstruction in the Wildcited by 10
  9. 2024PEAR: Phrase-Based Hand-Object Interaction Anticipationcited by 10
  10. 2025Mask2IV: Interaction-Centric Video Generation via Mask Trajectoriescited by 10
  11. 2024HOIST-Former: Hand-held Objects Identification, Segmentation, and Tracking in the Wildcited by 9
  12. 2024FoundHand: Large-Scale Domain-Specific Learning for Controllable Hand Image Generationcited by 8
30 more
  1. 2024GraspDiffusion: Synthesizing Realistic Whole-body Hand-Object Interactioncited by 7
  2. 2025UniHOPE: A Unified Approach for Hand-Only and Hand-Object Pose Estimationcited by 6
  3. 2024HandCraft: Anatomically Correct Restoration of Malformed Hands in Diffusion Generated Imagescited by 5
  4. 2024HANDI: Hand-Centric Text-and-Image Conditioned Video Generationcited by 5
  5. 2025Counting Hallucinations in Diffusion Modelscited by 5
  6. 2024Joint Diffusion for Universal Hand-Object Grasp Generationcited by 4
  7. 2025MIND-Edit: MLLM Insight-Driven Editing via Language-Vision Projectioncited by 4
  8. 2025FairHuman: Boosting Hand and Face Quality in Human Image Generation with Minimum Potential Delay Fairness in Diffusion Modelscited by 4
  9. 2025Conditioning Residuals for Diffusion Models via Representation Feedbackcited by 3
  10. 2025RODS: Robust Optimization Inspired Diffusion Sampling for Detecting and Reducing Hallucination in Generative Modelscited by 3
  11. 2025HandEval: Taking the First Step Towards Hand Quality Evaluation in Generated Imagescited by 3
  12. 2026Glove2Hand: Synthesizing Natural Hand-Object Interaction from Multi-Modal Sensing Glovescited by 3
  13. 2024Adaptive Multi-Modal Control of Digital Human Hand Synthesis Using a Region-Aware Cycle Losscited by 2
  14. 2025HanDrawer: Leveraging Spatial Information to Render Realistic Hands Using a Conditional Diffusion Model in Single Stagecited by 1
  15. 2025MGHanD: Multi-modal Guidance for authentic Hand Diffusioncited by 1
  16. 2025CROC: Evaluating and Training T2I Metrics with Pseudo- and Human-Labeled Contrastive Robustness Checkscited by 1
  17. 2026Beyond Language: Grounding Referring Expressions with Hand Pointing in Egocentric Visioncited by 1
  18. 2026THOM: Generating Physically Plausible Hand-Object Meshes From Textcited by 1
  19. 2026Local Intrinsic Dimension Unveils Hallucinations in Diffusion Modelscited by 1
  20. 2024Integrating Text-to-Music Models with Language Models: Composing Long Structured Music Piecescited by 0
  21. 2025Enhancing Monocular 3D Hand Reconstruction with Learned Texture Priorscited by 0
  22. 2026Generating metamers of human scene understandingcited by 0
  23. 2026Beyond Cropping and Rotation: Automated Evolution of Powerful Task-Specific Augmentations with Generative Modelscited by 0
  24. 2026SesaHand: Enhancing 3D Hand Reconstruction via Controllable Generation with Semantic and Structural Alignmentcited by 0
  25. 2026AlignVAR: Towards Globally Consistent Visual Autoregression for Image Super-Resolutioncited by 0
  26. 2026Na"ive PAINE: Lightweight Text-to-Image Generation Improvement with Prompt Evaluationcited by 0
  27. 2026T3HG-Editor: Text-driven 3D Human Garment Editing with Body Priors Embedded in SMPL-Xcited by 0
  28. 2026MixDiffusion: Mixing Diffusion-based Uni-condition Text-to-Image Generation Models for Multi-condition Image Synthesiscited by 0
  29. 2026Concept Guidance: Precise, Training-Free Latent Control for Text-to-Image Generationcited by 0
  30. 2026Mudragen: Geometrically Supervised Generation of Interacting Two-Hand Mudras for Preserving Indian Classical Dance Heritagecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Giving a Hand to Diffusion Models: a Two-Stage Approach to Improving Conditional Human Image Generation
  2. 2024Annotated Hands for Generative Models
  3. 2023Prompt-Propose-Verify: A Reliable Hand-Object-Interaction Data Generation Framework using Foundational Models
  4. 2026HandDreamer: Zero-Shot Text to 3D Hand Model Generation using Corrective Hand Shape Guidance
  5. 2023Affordance Diffusion: Synthesizing Hand-Object Interactions
  6. 2024RHanDS: Refining Malformed Hands for Generated Images with Decoupled Structure and Style Guidance
  7. 2024HOIDiffusion: Generating Realistic 3D Hand-Object Interaction Data
  8. 20253D Hand Mesh-Guided AI-Generated Malformed Hand Refinement with Hand Pose Transformation via Diffusion Model