mlsanityData to 7 Oct 2026

Affordance Diffusion: Synthesizing Hand-Object Interactions

arXiv 2303.12538 ↗First posted 21 Mar 2023cs.CV cs.ROcited by 110

On the map: Diffusion models → Diffusion and flow policies for robotics

Abstract (arXiv)

Recent successes in image synthesis are powered by large-scale diffusion models. However, most methods are currently limited to either text- or image-conditioned generation for synthesizing an entire image, texture transfer or inserting objects into a user-specified region. In contrast, in this work we focus on synthesizing complex interactions (ie, an articulated hand) with a given object. Given an RGB image of an object, we aim to hallucinate plausible images of a human hand interacting with it. We propose a two-step generative approach: a LayoutNet that samples an articulation-agnostic hand-object-interaction layout, and a ContentNet that synthesizes images of a hand grasping the object given the predicted layout. Both are built on top of a large-scale pretrained diffusion model to make use of its latent representation. Compared to baselines, the proposed method is shown to generalize better to novel objects and perform surprisingly well on out-of-distribution in-the-wild scenes of portable-sized objects. The resulting system allows us to predict descriptive affordance information, such as hand articulation and approaching orientation. Project page: https://judyye.github.io/affordiffusion-www

Key facts. Cited by 110 papers in the map (data through 7 Oct 2026). Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models. Builds on 70 of its 97 references that are in the map. First posted on arXiv 21 Mar 2023.

Builds on

70 of 97 references.

References · 70

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2020Denoising Diffusion Implicit Modelscited by 9,655
  5. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  6. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  7. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  8. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  9. 2022Classifier-Free Diffusion Guidancecited by 5,717
  10. 2022Generative Adversarial Networkscited by 5,603
  11. 2016Image-to-Image Translation with Conditional Adversarial Networkscited by 4,510
  12. 2013Auto-Encoding Variational Bayescited by 3,980
38 more
  1. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  2. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,286
  3. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,231
  4. 2021SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equationscited by 1,903
  5. 2015Spatial Transformer Networkscited by 1,423
  6. 2020DiffWave: A Versatile Diffusion Model for Audio Synthesiscited by 1,339
  7. 2021Palette: Image-to-Image Diffusion Modelscited by 1,296
  8. 2022Imagic: Text-Based Real Image Editing with Diffusion Modelscited by 1,172
  9. 2022Diffusion-LM Improves Controllable Text Generationcited by 1,006
  10. 2021StyleCLIP: Text-Driven Manipulation of StyleGAN Imagerycited by 990
  11. 2021Blended Diffusion for Text-driven Editing of Natural Imagescited by 924
  12. 2022eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoiserscited by 921
  13. 2018Scaling Egocentric Vision: The EPIC-KITCHENS Datasetcited by 789
  14. 2021Score-based Generative Modeling in Latent Spacecited by 634
  15. 2021CSDI: Conditional Score-based Diffusion Models for Probabilistic Time Series Imputationcited by 577
  16. 2022LION: Latent Point Diffusion Models for 3D Shape Generationcited by 552
  17. 20213D Shape Generation and Completion through Point-Voxel Diffusioncited by 516
  18. 2022GeoDiff: a Geometric Diffusion Model for Molecular Conformation Generationcited by 511
  19. 2022StyleGAN-XL: Scaling StyleGAN to Large Diverse Datasetscited by 489
  20. 2019Learning joint reconstruction of hands and manipulated objectscited by 377
  21. 2022HOI4D: A 4D Egocentric Dataset for Category-Level Human-Object Interactioncited by 321
  22. 2018Everybody Dance Nowcited by 313
  23. 2022Text2LIVE: Text-Driven Layered Image and Video Editingcited by 312
  24. 2020Understanding Human Hands in Contact at Internet Scalecited by 281
  25. 2017First-Person Hand Action Benchmark with RGB-D Videos and 3D Hand Pose Annotationscited by 229
  26. 2021Anticipative Video Transformercited by 197
  27. 2021DexMV: Imitation Learning for Dexterous Manipulation from Human Videoscited by 194
  28. 2020ContactPose: A Dataset of Grasps with Object Contact and Hand Posecited by 180
  29. 2021Where2Act: From Pixels to Actions for Articulated 3D Objectscited by 179
  30. 2020Grasping Field: Learning Implicit Representations for Human Graspscited by 178
  31. 2021Hand-Object Contact Consistency Reasoning for Human Grasps Generationcited by 160
  32. 2022Human-to-Robot Imitation in the Wildcited by 157
  33. 2017GANerated Hands for Real-time 3D Hand Tracking from Monocular RGBcited by 147
  34. 2019H+O: Unified Egocentric Recognition of 3D Hand-Object Poses and Interactionscited by 139
  35. 2021ContactOpt: Optimizing Contact to Improve Graspscited by 137
  36. 2018Grounded Human-Object Interaction Hotspots from Videocited by 128
  37. 2019Generating 3D People in Scenes without Peoplecited by 127
  38. 2020EGO-TOPO: Environment Affordances from Egocentric Videocited by 111

50 most cited of 70 All 70 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 15; 2024: 41; 2025: 33; 2026: 21.

110 papers.

Sort:

By area:

  1. 2023InterDiff: Generating 3D Human-Object Interactions with Physics-Informed Diffusioncited by 181
  2. 2023CG-HOI: Contact-Guided 3D Human-Object Interaction Generationcited by 137
  3. 2024TACO: Benchmarking Generalizable Bimanual Tool-ACtion-Object Understandingcited by 85
  4. 2023An Outlook into the Future of Egocentric Visioncited by 84
  5. 2023Diffusion-Guided Reconstruction of Everyday Hand-Object Interaction Clipscited by 73
  6. 2024Robo-ABC: Affordance Generalization Beyond Categories via Semantic Correspondence for Robot Manipulationcited by 69
  7. 2025InterMimic: Towards Universal Whole-Body Control for Physics-Based Human-Object Interactionscited by 67
  8. 2024InterDreamer: Zero-Shot Text to 3D Dynamic Human-Object Interactioncited by 63
  9. 2023ReMoS: 3D Motion-Conditioned Reaction Synthesis for Two-Person Interactionscited by 62
  10. 2024DiffH2O: Diffusion-Based Synthesis of Hand-Object Interactions from Textual Descriptionscited by 60
  11. 2024GraspXL: Generating Grasping Motions for Diverse Objects at Scalecited by 59
  12. 2024AffordanceLLM: Grounding Affordance from Vision Language Modelscited by 55
38 more
  1. 2023Towards Generalizable Zero-Shot Manipulation via Translating Human Interaction Planscited by 51
  2. 2024Plan-Seq-Learn: Language Model Guided RL for Solving Long Horizon Robotics Taskscited by 51
  3. 2024G-HOP: Generative Hand-Object Prior for Interaction Reconstruction and Grasp Synthesiscited by 43
  4. 2023HandRefiner: Refining Malformed Hands in Generated Images by Diffusion-based Conditional Inpaintingcited by 42
  5. 2023GenZI: Zero-Shot 3D Human-Scene Interaction Generationcited by 40
  6. 2024HRP: Human Affordances for Robotic Pre-Trainingcited by 40
  7. 2024HOIDiffusion: Generating Realistic 3D Hand-Object Interaction Datacited by 37
  8. 2023Dexterous Functional Graspingcited by 36
  9. 2024Diffusion Meets DAgger: Supercharging Eye-in-hand Imitation Learningcited by 34
  10. 2024EasyHOI: Unleashing the Power of Large Models for Reconstructing Hand-Object Interactions in the Wildcited by 33
  11. 2024Beyond the Contact: Discovering Comprehensive Affordance for 3D Objects from Pre-trained 2D Diffusion Modelscited by 32
  12. 2024Put Myself in Your Shoes: Lifting the Egocentric Perspective from Exocentric Videoscited by 32
  13. 2025ZeroMimic: Distilling Robotic Manipulation Skills from Web Videoscited by 31
  14. 2025InterAct: Advancing Large-Scale Versatile 3D Human-Object Interaction Generationcited by 31
  15. 2025HUMOTO: A 4D Dataset of Mocap Human Object Interactionscited by 30
  16. 2023LEGO: Learning EGOcentric Action Frame Generation via Visual Instruction Tuningcited by 28
  17. 2024InterTrack: Tracking Human Object Interaction without Object Templatescited by 22
  18. 2023Look Ma, No Hands! Agent-Environment Factorization of Egocentric Videoscited by 21
  19. 2024Diff-IP2D: Diffusion-Based Hand-Object Interaction Prediction on Egocentric Videoscited by 21
  20. 2024HOI-Swap: Swapping Objects in Videos with Hand-Object Interaction Awarenesscited by 21
  21. 2024Controlling the World by Sleight of Handcited by 19
  22. 2025Re-HOLD: Video Hand Object Interaction Reenactment via adaptive Layout-instructed Diffusion Modelcited by 19
  23. 2025DreamActor-H1: High-Fidelity Human-Product Demonstration Video Generation via Motion-designed Diffusion Transformerscited by 19
  24. 2024Generative Image as Action Modelscited by 18
  25. 2024ManiVideo: Generating Hand-Object Manipulation Video with Dexterous and Generalizable Graspingcited by 18
  26. 2024MADiff: Motion-Aware Mamba Diffusion Models for Hand Trajectory Prediction on Egocentric Videoscited by 17
  27. 2025The Invisible EgoHand: 3D Hand Forecasting through EgoBody Pose Estimationcited by 16
  28. 2025Simulating the Visual World with Artificial Intelligence: A Roadmapcited by 15
  29. 2023Exploiting Multimodal Synthetic Data for Egocentric Human-Object Interaction Detection in an Industrial Scenariocited by 14
  30. 2024HandBooster: Boosting 3D Hand-Mesh Reconstruction by Conditional Synthesis and Sampling of Hand-Object Interactionscited by 13
  31. 2026LOME: Learning Human-Object Manipulation with Action-Conditioned Egocentric World Modelcited by 12
  32. 2024ManiFoundation Model for General-Purpose Robotic Manipulation of Contact Synthesis with Arbitrary Objects and Robotscited by 11
  33. 2024InterDyn: Controllable Interactive Dynamics with Video Diffusion Modelscited by 11
  34. 2023Are Synthetic Data Useful for Egocentric Hand-Object Interaction Detection?cited by 10
  35. 2024AttentionHand: Text-driven Controllable Hand Image Generation for 3D Hand Reconstruction in the Wildcited by 10
  36. 2024RealisHuman: A Two-Stage Approach for Refining Malformed Human Parts in Generated Imagescited by 10
  37. 2025How Do I Do That? Synthesizing 3D Hand Motion and Contacts for Everyday Interactionscited by 10
  38. 2025Web2Grasp: Learning Functional Grasps from Web Images of Hand-Object Interactionscited by 10

50 most cited of 110 All 110 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Prompt-Propose-Verify: A Reliable Hand-Object-Interaction Data Generation Framework using Foundational Models
  2. 2026FSAG: Enhancing Human-to-Dexterous-Hand Finger-Specific Affordance Grounding via Diffusion Models
  3. 2024Giving a Hand to Diffusion Models: a Two-Stage Approach to Improving Conditional Human Image Generation
  4. 2025SIGHT: Synthesizing Image-Text Conditioned and Geometry-Guided 3D Hand-Object Trajectories
  5. 2024InterHandGen: Two-Hand Interaction Generation via Cascaded Reverse Diffusion
  6. 2026GIRAF: Towards Generalizable Human Interactions with Articulated Objects
  7. 2024HanDiffuser: Text-to-Image Generation With Realistic Hand Appearances
  8. 2026PhotoHOI: Synthesizing 3D Hand-Object Interactions from a Single RGB Photograph