mlsanityData to 14 Oct 2026

HOIDiffusion: Generating Realistic 3D Hand-Object Interaction Data

arXiv 2403.12011 ↗First posted 18 Mar 2024cs.CVcited by 37

On the map: Diffusion models → Diffusion and flow policies for robotics

Abstract (arXiv)

3D hand-object interaction data is scarce due to the hardware constraints in scaling up the data collection process. In this paper, we propose HOIDiffusion for generating realistic and diverse 3D hand-object interaction data. Our model is a conditional diffusion model that takes both the 3D hand-object geometric structure and text description as inputs for image synthesis. This offers a more controllable and realistic synthesis as we can specify the structure and style inputs in a disentangled manner. HOIDiffusion is trained by leveraging a diffusion model pre-trained on large-scale natural images and a few 3D human demonstrations. Beyond controllable image synthesis, we adopt the generated 3D data for learning 6D object pose estimation and show its effectiveness in improving perception systems. Project page: https://mq-zhang1.github.io/HOIDiffusion

Key facts. Cited by 37 papers in the map (data through 14 Oct 2026). Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models. Builds on 52 of its 67 references that are in the map. First posted on arXiv 18 Mar 2024.

Builds on

52 of 67 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Affordance Diffusion: Synthesizing Hand-Object Interactions21 Mar

Other references · 51

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  4. 2020Denoising Diffusion Implicit Modelscited by 9,781
  5. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,255
  6. 2023Visual Instruction Tuningcited by 7,869
  7. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  8. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  9. 2022Classifier-Free Diffusion Guidancecited by 5,847
  10. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,503
  11. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,761
  12. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,614
38 more
  1. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,300
  2. 2016Improved Techniques for Training GANscited by 2,970
  3. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,424
  4. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,245
  5. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,556
  6. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,555
  7. 2019Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfercited by 1,495
  8. 2023T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Modelscited by 1,351
  9. 2018Scaling Egocentric Vision: The EPIC-KITCHENS Datasetcited by 795
  10. 2019Normalized Object Coordinate Space for Category-Level 6D Object Pose and Size Estimationcited by 495
  11. 2020GRAB: A Dataset of Whole-Body Human Grasping of Objectscited by 405
  12. 2019Learning joint reconstruction of hands and manipulated objectscited by 378
  13. 2022HOI4D: A 4D Egocentric Dataset for Category-Level Human-Object Interactioncited by 327
  14. 2021DexYCB: A Benchmark for Capturing Hand Grasping of Objectscited by 325
  15. 2022ARCTIC: A Dataset for Dexterous Bimanual Hand-Object Manipulationcited by 312
  16. 2020Understanding Human Hands in Contact at Internet Scalecited by 281
  17. 2021Generating Images with Sparse Representationscited by 266
  18. 2021DexMV: Imitation Learning for Dexterous Manipulation from Human Videoscited by 206
  19. 2020ContactPose: A Dataset of Grasps with Object Contact and Hand Posecited by 183
  20. 2020Grasping Field: Learning Implicit Representations for Human Graspscited by 179
  21. 2021FrankMocap: A Monocular 3D Whole-Body Pose Estimation System via Regression and Integrationcited by 169
  22. 2020Shape Prior Deformation for Categorical 6D Object Pose and Size Estimationcited by 161
  23. 2021Hand-Object Contact Consistency Reasoning for Human Grasps Generationcited by 160
  24. 2020Reconstructing Hand-Object Interactions in the Wildcited by 144
  25. 2021GOAL: Generating 4D Whole-Body Motion for Hand-Object Graspingcited by 144
  26. 2021Semi-Supervised 3D Hand-Object Poses Estimation with Interactions in Timecited by 141
  27. 2020Leveraging Photometric Consistency over Time for Sparsely Supervised Hand-Object Reconstructioncited by 139
  28. 2021ContactOpt: Optimizing Contact to Improve Graspscited by 139
  29. 2022OakInk: A Large-scale Knowledge Repository for Understanding Hand-Object Interactioncited by 131
  30. 2023DatasetDM: Synthesizing Data with Perception Annotations Using Diffusion Modelscited by 123
  31. 2021A-SDF: Learning Disentangled Signed Distance Functions for Articulated Shape Representationcited by 114
  32. 2021D-Grasp: Physically Plausible Dynamic Grasp Synthesis for Hand-Object Interactionscited by 113
  33. 2020HOPE-Net: A Graph-based Model for Hand-Object Pose Estimationcited by 112
  34. 2019Efficient Learning on Point Clouds with Basis Point Setscited by 103
  35. 2021DualPoseNet: Category-level 6D Object Pose and Size Estimation Using Dual Pose Network with Refined Learning of Pose Consistencycited by 103
  36. 2019ContactGrasp: Functional Multi-finger Grasp Synthesis from Contactcited by 87
  37. 2016Real-time Joint Tracking of a Hand Manipulating an Object from RGB-D Inputcited by 56
  38. 2019Generalized Feedback Loop for Joint Hand-Object Pose Estimationcited by 22

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 12; 2025: 12; 2026: 13.

37 papers.

Sort:

By area:

  1. 2025A Survey on Human Interaction Motion Generationcited by 25
  2. 2024Large Language Models for Manufacturingcited by 24
  3. 2024InterTrack: Tracking Human Object Interaction without Object Templatescited by 23
  4. 2025DreamActor-H1: High-Fidelity Human-Product Demonstration Video Generation via Motion-designed Diffusion Transformerscited by 22
  5. 2024Diff-IP2D: Diffusion-Based Hand-Object Interaction Prediction on Egocentric Videoscited by 21
  6. 2024HOI-Swap: Swapping Objects in Videos with Hand-Object Interaction Awarenesscited by 21
  7. 2025Re-HOLD: Video Hand Object Interaction Reenactment via adaptive Layout-instructed Diffusion Modelcited by 19
  8. 2024MADiff: Motion-Aware Mamba Diffusion Models for Hand Trajectory Prediction on Egocentric Videoscited by 18
  9. 2026LOME: Learning Human-Object Manipulation with Action-Conditioned Egocentric World Modelcited by 13
  10. 2024InterDyn: Controllable Interactive Dynamics with Video Diffusion Modelscited by 12
  11. 2025Generating 6DoF Object Manipulation Trajectories from Action Description in Egocentric Visioncited by 11
  12. 2024Human Action Anticipation: A Surveycited by 9
25 more
  1. 2025EgoWorld: Translating Exocentric View to Egocentric View using Rich Exocentric Observationscited by 9
  2. 2024GraspDiffusion: Synthesizing Realistic Whole-body Hand-Object Interactioncited by 8
  3. 2025Novel Diffusion Models for Multimodal 3D Hand Trajectory Predictioncited by 8
  4. 2024Robust 3D Shape Reconstruction in Zero-Shot from a Single Image in the Wildcited by 7
  5. 2026HVG-3D: Bridging Real and Simulation Domains for 3D-Conditional Hand-Object Interaction Video Synthesiscited by 7
  6. 20243D Whole-body Grasp Synthesis with Directional Controllabilitycited by 6
  7. 2025UniHOPE: A Unified Approach for Hand-Only and Hand-Object Pose Estimationcited by 6
  8. 2025iDiT-HOI: Inpainting-based Hand Object Interaction Reenactment via Video Diffusion Transformercited by 5
  9. 2025Uni-Hand: Universal Hand Motion Forecasting in Egocentric Viewscited by 5
  10. 2024Joint Diffusion for Universal Hand-Object Grasp Generationcited by 4
  11. 2025What Are You Doing? A Closer Look at Controllable Human Video Generationcited by 3
  12. 2026Glove2Hand: Synthesizing Natural Hand-Object Interaction from Multi-Modal Sensing Glovescited by 3
  13. 2026PAM: A Pose-Appearance-Motion Engine for Sim-to-Real HOI Video Generationcited by 3
  14. 2025EgoLoc: A Generalizable Solution for Temporal Interaction Localization in Egocentric Videoscited by 1
  15. 2025DexCanvas: Bridging Human Demonstrations and Robot Learning for Dexterous Manipulationcited by 1
  16. 2026GenHOI: Towards Object-Consistent Hand-Object Interaction with Temporally Balanced and Spatially Selective Object Injectioncited by 1
  17. 2026GenHOI: Generalized Hand-Object Pose Estimation with Occlusion Awarenesscited by 1
  18. 2026THOM: Generating Physically Plausible Hand-Object Meshes From Textcited by 1
  19. 2026Hand-Object Interaction in the Age of Large Foundation Models:Reconstruction, Generation, and Embodied Transfercited by 1
  20. 2024GenHeld: Generating and Editing Handheld Objectscited by 0
  21. 2026EgoFlow: Gradient-Guided Flow Matching for Egocentric 6DoF Object Motion Generationcited by 0
  22. 2026DCGrasp: Distance-aware Controllable Grasp Generationcited by 0
  23. 2026MixDiffusion: Mixing Diffusion-based Uni-condition Text-to-Image Generation Models for Multi-condition Image Synthesiscited by 0
  24. 2026GraspHOI: Full-Body 3D Human-Object Reconstruction with Finger-Level Grasps from a Single In-the-Wild Imagecited by 0
  25. 2026GazeDiT: Gaze-Accurate Diffusion Image Generation for Eye Tracking via Spatial Conditioningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024DiffH2O: Diffusion-Based Synthesis of Hand-Object Interactions from Textual Descriptions
  2. 2023Prompt-Propose-Verify: A Reliable Hand-Object-Interaction Data Generation Framework using Foundational Models
  3. 2025HOIDiNi: Human-Object Interaction through Diffusion Noise Optimization
  4. 2025SIGHT: Synthesizing Image-Text Conditioned and Geometry-Guided 3D Hand-Object Trajectories
  5. 2026HO-Flow: Generalizable Hand-Object Interaction Generation with Latent Flow Matching
  6. 2024A Versatile and Differentiable Hand-Object Interaction Representation
  7. 2024Text2HOI: Text-guided 3D Motion Generation for Hand-Object Interaction
  8. 2025Follow My Hold: Hand-Object Interaction Reconstruction through Geometric Guidance