# HanDiffuser: Text-to-Image Generation With Realistic Hand Appearances

arXiv: https://arxiv.org/abs/2403.01693 · HTML: https://mlsanity.com/paper/2403.01693/

## Key facts

- Cited by 42 papers in the map (data through 14 Oct 2026).
- Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models.
- Builds on 60 of its 78 references that are in the map.
- First posted on arXiv 4 Mar 2024.

## Placed in

- Diffusion models / [Diffusion and flow policies for robotics](https://mlsanity.com/diffusion/diffusion-and-flow-policies-for-robotics/)

## Builds on (most cited first, 30 of 60)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models](https://mlsanity.com/paper/2112.10741/) (2021)
- [SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations](https://mlsanity.com/paper/2108.01073/) (2021)
- [AMASS: Archive of Motion Capture as Surface Shapes](https://mlsanity.com/paper/1904.03278/) (2019)
- [Cascaded Diffusion Models for High Fidelity Image Generation](https://mlsanity.com/paper/2106.15282/) (2021)
- [OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields](https://mlsanity.com/paper/1812.08008/) (2018)
- [Human Motion Diffusion Model](https://mlsanity.com/paper/2209.14916/) (2022)
- [MotionDiffuse: Text-Driven Human Motion Generation with Diffusion Model](https://mlsanity.com/paper/2208.15001/) (2022)
- [Vector Quantized Diffusion Model for Text-to-Image Synthesis](https://mlsanity.com/paper/2111.14822/) (2021)
- [Pseudo Numerical Methods for Diffusion Models on Manifolds](https://mlsanity.com/paper/2202.09778/) (2022)
- [AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networks](https://mlsanity.com/paper/1711.10485/) (2017)
- [NTU RGB+D 120: A Large-Scale Benchmark for 3D Human Activity Understanding](https://mlsanity.com/paper/1905.04757/) (2019)
- [Action-Conditioned 3D Human Motion Synthesis with Transformer VAE](https://mlsanity.com/paper/2104.05670/) (2021)
- [TEMOS: Generating diverse human motions from textual descriptions](https://mlsanity.com/paper/2204.14109/) (2022)
- [Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priors](https://mlsanity.com/paper/2203.13131/) (2022)
- [Pix2seq: A Language Modeling Framework for Object Detection](https://mlsanity.com/paper/2109.10852/) (2021)
- [The KIT Motion-Language Dataset](https://mlsanity.com/paper/1607.03827/) (2016)
- [ARCTIC: A Dataset for Dexterous Bimanual Hand-Object Manipulation](https://mlsanity.com/paper/2204.13662/) (2022)
- [DM-GAN: Dynamic Memory Generative Adversarial Networks for Text-to-Image Synthesis](https://mlsanity.com/paper/1904.01310/) (2019)

## Built on by (most cited first, 30 of 42)

- [Understanding Hallucinations in Diffusion Models through Mode Interpolation](https://mlsanity.com/paper/2406.09358/) (2024)
- [ManiVideo: Generating Hand-Object Manipulation Video with Dexterous and Generalizable Grasping](https://mlsanity.com/paper/2412.16212/) (2024)
- [MoLE: Enhancing Human-centric Text-to-image Diffusion via Mixture of Low-rank Experts](https://mlsanity.com/paper/2410.23332/) (2024)
- [X-Dancer: Expressive Music to Human Dance Video Generation](https://mlsanity.com/paper/2502.17414/) (2025)
- [Detecting Human Artifacts from Text-to-Image Models](https://mlsanity.com/paper/2411.13842/) (2024)
- [Hand1000: Generating Realistic Hands from Text with Only 1,000 Images](https://mlsanity.com/paper/2408.15461/) (2024)
- [Mitigating Hallucinations in Diffusion Models through Adaptive Attention Modulation](https://mlsanity.com/paper/2502.16872/) (2025)
- [AttentionHand: Text-driven Controllable Hand Image Generation for 3D Hand Reconstruction in the Wild](https://mlsanity.com/paper/2407.18034/) (2024)
- [PEAR: Phrase-Based Hand-Object Interaction Anticipation](https://mlsanity.com/paper/2407.21510/) (2024)
- [Mask2IV: Interaction-Centric Video Generation via Mask Trajectories](https://mlsanity.com/paper/2510.03135/) (2025)
- [HOIST-Former: Hand-held Objects Identification, Segmentation, and Tracking in the Wild](https://mlsanity.com/paper/2404.13819/) (2024)
- [GraspDiffusion: Synthesizing Realistic Whole-body Hand-Object Interaction](https://mlsanity.com/paper/2410.13911/) (2024)
- [FoundHand: Large-Scale Domain-Specific Learning for Controllable Hand Image Generation](https://mlsanity.com/paper/2412.02690/) (2024)
- [UniHOPE: A Unified Approach for Hand-Only and Hand-Object Pose Estimation](https://mlsanity.com/paper/2503.13303/) (2025)
- [Counting Hallucinations in Diffusion Models](https://mlsanity.com/paper/2510.13080/) (2025)
- [HandCraft: Anatomically Correct Restoration of Malformed Hands in Diffusion Generated Images](https://mlsanity.com/paper/2411.04332/) (2024)
- [HANDI: Hand-Centric Text-and-Image Conditioned Video Generation](https://mlsanity.com/paper/2412.04189/) (2024)
- [Joint Diffusion for Universal Hand-Object Grasp Generation](https://mlsanity.com/paper/2409.04560/) (2024)
- [MIND-Edit: MLLM Insight-Driven Editing via Language-Vision Projection](https://mlsanity.com/paper/2505.19149/) (2025)
- [FairHuman: Boosting Hand and Face Quality in Human Image Generation with Minimum Potential Delay Fairness in Diffusion Models](https://mlsanity.com/paper/2507.02714/) (2025)
- [Conditioning Residuals for Diffusion Models via Representation Feedback](https://mlsanity.com/paper/2505.10999/) (2025)
- [RODS: Robust Optimization Inspired Diffusion Sampling for Detecting and Reducing Hallucination in Generative Models](https://mlsanity.com/paper/2507.12201/) (2025)
- [HandEval: Taking the First Step Towards Hand Quality Evaluation in Generated Images](https://mlsanity.com/paper/2510.08978/) (2025)
- [Glove2Hand: Synthesizing Natural Hand-Object Interaction from Multi-Modal Sensing Gloves](https://mlsanity.com/paper/2603.20850/) (2026)
- [Adaptive Multi-Modal Control of Digital Human Hand Synthesis Using a Region-Aware Cycle Loss](https://mlsanity.com/paper/2409.09149/) (2024)
- [CROC: Evaluating and Training T2I Metrics with Pseudo- and Human-Labeled Contrastive Robustness Checks](https://mlsanity.com/paper/2505.11314/) (2025)
- [Beyond Language: Grounding Referring Expressions with Hand Pointing in Egocentric Vision](https://mlsanity.com/paper/2603.26646/) (2026)
- [HanDrawer: Leveraging Spatial Information to Render Realistic Hands Using a Conditional Diffusion Model in Single Stage](https://mlsanity.com/paper/2503.02127/) (2025)
- [MGHanD: Multi-modal Guidance for authentic Hand Diffusion](https://mlsanity.com/paper/2503.08133/) (2025)
- [THOM: Generating Physically Plausible Hand-Object Meshes From Text](https://mlsanity.com/paper/2604.02736/) (2026)

## Abstract (arXiv)

Text-to-image generative models can generate high-quality humans, but realism is lost when generating hands. Common artifacts include irregular hand poses, shapes, incorrect numbers of fingers, and physically implausible finger orientations. To generate images with realistic hands, we propose a novel diffusion-based architecture called HanDiffuser that achieves realism by injecting hand embeddings in the generative process. HanDiffuser consists of two components: a Text-to-Hand-Params diffusion model to generate SMPL-Body and MANO-Hand parameters from input text prompts, and a Text-Guided Hand-Params-to-Image diffusion model to synthesize images by conditioning on the prompts and hand parameters generated by the previous component. We incorporate multiple aspects of hand representation, including 3D shapes and joint-level finger positions, orientations and articulations, for robust learning and reliable performance during inference. We conduct extensive quantitative and qualitative experiments and perform user studies to demonstrate the efficacy of our method in generating images with high-quality hands.