# Beyond the Contact: Discovering Comprehensive Affordance for 3D Objects from Pre-trained 2D Diffusion Models

arXiv: https://arxiv.org/abs/2401.12978 · HTML: https://mlsanity.com/paper/2401.12978/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- Placed in Video generation, one of 14 directions in Diffusion models.
- Builds on 75 of its 98 references that are in the map.
- First posted on arXiv 23 Jan 2024.

## Placed in

- Diffusion models / [Video generation](https://mlsanity.com/diffusion/video-generation/)

## Builds on (most cited first, 30 of 75)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection](https://mlsanity.com/paper/2303.05499/) (2023)
- [ShapeNet: An Information-Rich 3D Model Repository](https://mlsanity.com/paper/1512.03012/) (2015)
- [Prompt-to-Prompt Image Editing with Cross Attention Control](https://mlsanity.com/paper/2208.01626/) (2022)
- [SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations](https://mlsanity.com/paper/2108.01073/) (2021)
- [Expressive Body Capture: 3D Hands, Face, and Body from a Single Image](https://mlsanity.com/paper/1904.05866/) (2019)
- [Null-text Inversion for Editing Real Images using Guided Diffusion Models](https://mlsanity.com/paper/2211.09794/) (2022)
- [OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields](https://mlsanity.com/paper/1812.08008/) (2018)
- [SAPIEN: A SimulAted Part-based Interactive ENvironment](https://mlsanity.com/paper/2003.08515/) (2020)
- [PartNet: A Large-scale Benchmark for Fine-grained and Hierarchical Part-level 3D Object Understanding](https://mlsanity.com/paper/1812.02713/) (2018)
- [Deep Marching Tetrahedra: a Hybrid Representation for High-Resolution 3D Shape Synthesis](https://mlsanity.com/paper/2111.04276/) (2021)
- [PointRend: Image Segmentation as Rendering](https://mlsanity.com/paper/1912.08193/) (2019)
- [GRAB: A Dataset of Whole-Body Human Grasping of Objects](https://mlsanity.com/paper/2008.11200/) (2020)
- [Is synthetic data from generative models ready for image recognition?](https://mlsanity.com/paper/2210.07574/) (2022)
- [TEXTure: Text-Guided Texturing of 3D Shapes](https://mlsanity.com/paper/2302.01721/) (2023)
- [Resolving 3D Human Pose Ambiguities with 3D Scene Constraints](https://mlsanity.com/paper/1908.06963/) (2019)
- [Effective Data Augmentation With Diffusion Models](https://mlsanity.com/paper/2302.07944/) (2023)
- [DatasetGAN: Efficient Labeled Data Factory with Minimal Human Effort](https://mlsanity.com/paper/2104.06490/) (2021)
- [BEHAVE: Dataset and Method for Tracking Human Object Interactions](https://mlsanity.com/paper/2204.06950/) (2022)
- [Affordances from Human Videos as a Versatile Representation for Robotics](https://mlsanity.com/paper/2304.08488/) (2023)
- [Long-term Human Motion Prediction with Scene Context](https://mlsanity.com/paper/2007.03672/) (2020)
- [ContactPose: A Dataset of Grasps with Object Contact and Hand Pose](https://mlsanity.com/paper/2007.09545/) (2020)
- [InterDiff: Generating 3D Human-Object Interactions with Physics-Informed Diffusion](https://mlsanity.com/paper/2308.16905/) (2023)

## Built on by (most cited first, 30 of 32)

- [InterMimic: Towards Universal Whole-Body Control for Physics-Based Human-Object Interactions](https://mlsanity.com/paper/2502.20390/) (2025)
- [ZeroHSI: Zero-Shot 4D Human-Scene Interaction by Video Generation](https://mlsanity.com/paper/2412.18600/) (2024)
- [InterTrack: Tracking Human Object Interaction without Object Templates](https://mlsanity.com/paper/2408.13953/) (2024)
- [PICO: Reconstructing 3D People In Contact with Objects](https://mlsanity.com/paper/2504.17695/) (2025)
- [TriDi: Trilateral Diffusion of 3D Humans, Objects, and Interactions](https://mlsanity.com/paper/2412.06334/) (2024)
- [InteractAnything: Zero-shot Human Object Interaction Synthesis via LLM Feedback and Object Affordance Parsing](https://mlsanity.com/paper/2505.24315/) (2025)
- [InterDyn: Controllable Interactive Dynamics with Video Diffusion Models](https://mlsanity.com/paper/2412.11785/) (2024)
- [Weakly-Supervised Affordance Grounding Guided by Part-Level Semantic Priors](https://mlsanity.com/paper/2505.24103/) (2025)
- [VolumetricSMPL: A Neural Volumetric Body Model for Efficient Interactions, Contacts, and Collisions](https://mlsanity.com/paper/2506.23236/) (2025)
- [GraspDiffusion: Synthesizing Realistic Whole-body Hand-Object Interaction](https://mlsanity.com/paper/2410.13911/) (2024)
- [Target-Aware Video Diffusion Models](https://mlsanity.com/paper/2503.18950/) (2025)
- [Zero-Shot Human-Object Interaction Synthesis with Multimodal Priors](https://mlsanity.com/paper/2503.20118/) (2025)
- [HOI-PAGE: Zero-Shot Human-Object Interaction Generation with Part Affordance Guidance](https://mlsanity.com/paper/2506.07209/) (2025)
- [Affogato: Open-Vocabulary Affordance Grounding with Automated Data Generation at Scale](https://mlsanity.com/paper/2506.12009/) (2025)
- [GenHSI: Controllable Generation of Human-Scene Interaction Videos](https://mlsanity.com/paper/2506.19840/) (2025)
- [Diffusion Models are Open-World Affordance Learners: Leveraging Generative Priors for 3D Affordance Learning](https://mlsanity.com/paper/2508.01651/) (2025)
- [Learning 3D Object Spatial Relationships from Pre-trained 2D Diffusion Models](https://mlsanity.com/paper/2503.19914/) (2025)
- [DAViD: Modeling Dynamic Affordance of 3D Objects Using Pre-trained Video Diffusion Models](https://mlsanity.com/paper/2501.08333/) (2025)
- [SyncSDE: A Probabilistic Framework for Diffusion Synchronization](https://mlsanity.com/paper/2503.21555/) (2025)
- [PA-HOI: A Physics-Aware Human and Object Interaction Dataset](https://mlsanity.com/paper/2508.06205/) (2025)
- [Learning to Generate Human-Human-Object Interactions from Textual Descriptions](https://mlsanity.com/paper/2511.20446/) (2025)
- [HOPE: Hand-Object Pressure Estimation from Monocular Videos](https://mlsanity.com/paper/2608.06192/) (2026)
- [H2OFlow: Grounding Human-Object Affordances with 3D Generative Models and Dense Diffused Flows](https://mlsanity.com/paper/2510.21769/) (2025)
- [TeHOR: Text-Guided 3D Human and Object Reconstruction with Textures](https://mlsanity.com/paper/2602.19679/) (2026)
- [HAMMER: Harnessing MLLM via Cross-Modal Integration for Intention-Driven 3D Affordance Grounding](https://mlsanity.com/paper/2603.02329/) (2026)
- [CompassAD: Intent-Driven 3D Affordance Grounding in Functionally Competing Objects](https://mlsanity.com/paper/2604.02060/) (2026)
- [InHabit: Leveraging Image Foundation Models for Scalable 3D Human Placement](https://mlsanity.com/paper/2604.19673/) (2026)
- [Analogical Trajectory Transfer](https://mlsanity.com/paper/2605.14393/) (2026)
- [Towards Affordance-Aware Articulation Synthesis for Rigged Objects](https://mlsanity.com/paper/2501.12393/) (2025)
- [Probing and Bridging Geometry-Interaction Cues for Affordance Reasoning in Vision Foundation Models](https://mlsanity.com/paper/2602.20501/) (2026)

## Abstract (arXiv)

Understanding the inherent human knowledge in interacting with a given environment (e.g., affordance) is essential for improving AI to better assist humans. While existing approaches primarily focus on human-object contacts during interactions, such affordance representation cannot fully address other important aspects of human-object interactions (HOIs), i.e., patterns of relative positions and orientations. In this paper, we introduce a novel affordance representation, named Comprehensive Affordance (ComA). Given a 3D object mesh, ComA models the distribution of relative orientation and proximity of vertices in interacting human meshes, capturing plausible patterns of contact, relative orientations, and spatial relationships. To construct the distribution, we present a novel pipeline that synthesizes diverse and realistic 3D HOI samples given any 3D object mesh. The pipeline leverages a pre-trained 2D inpainting diffusion model to generate HOI images from object renderings and lifts them into 3D. To avoid the generation of false affordances, we propose a new inpainting framework, Adaptive Mask Inpainting. Since ComA is built on synthetic samples, it can extend to any object in an unbounded manner. Through extensive experiments, we demonstrate that ComA outperforms competitors that rely on human annotations in modeling contact-based affordance. Importantly, we also showcase the potential of ComA to reconstruct human-object interactions in 3D through an optimization framework, highlighting its advantage in incorporating both contact and non-contact properties.