# Diffusion Models are Open-World Affordance Learners: Leveraging Generative Priors for 3D Affordance Learning

arXiv: https://arxiv.org/abs/2508.01651 · HTML: https://mlsanity.com/paper/2508.01651/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Placed in Video generation, one of 14 directions in Diffusion models.
- Builds on 51 of its 75 references that are in the map.
- First posted on arXiv 3 Aug 2025.

## Placed in

- Diffusion models / [Video generation](https://mlsanity.com/diffusion/video-generation/)

## Builds on (most cited first, 30 of 51)

- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation](https://mlsanity.com/paper/1606.04797/) (2016)
- [LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairs](https://mlsanity.com/paper/2111.02114/) (2021)
- [eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoisers](https://mlsanity.com/paper/2211.01324/) (2022)
- [Label-Efficient Semantic Segmentation with Diffusion Models](https://mlsanity.com/paper/2112.03126/) (2021)
- [Emergent Correspondence from Image Diffusion](https://mlsanity.com/paper/2306.03881/) (2023)
- [Open-Vocabulary Panoptic Segmentation with Text-to-Image Diffusion Models](https://mlsanity.com/paper/2303.04803/) (2023)
- [DiffusionDet: Diffusion Model for Object Detection](https://mlsanity.com/paper/2211.09788/) (2022)
- [Your Diffusion Model is Secretly a Zero-Shot Classifier](https://mlsanity.com/paper/2303.16203/) (2023)
- [GRES: Generalized Referring Expression Segmentation](https://mlsanity.com/paper/2306.00968/) (2023)
- [CrossPoint: Self-Supervised Cross-Modal Contrastive Learning for 3D Point Cloud Understanding](https://mlsanity.com/paper/2203.00680/) (2022)
- [PointFusion: Deep Sensor Fusion for 3D Bounding Box Estimation](https://mlsanity.com/paper/1711.10871/) (2017)
- [Contrast with Reconstruct: Contrastive 3D Representation Learning Guided by Generative Pretraining](https://mlsanity.com/paper/2302.02318/) (2023)
- [ManipLLM: Embodied Multimodal Large Language Model for Object-Centric Robotic Manipulation](https://mlsanity.com/paper/2312.16217/) (2023)
- [Referring Transformer: A One-step Approach to Multi-task Visual Grounding](https://mlsanity.com/paper/2106.03089/) (2021)
- [RoboBrain: A Unified Brain Model for Robotic Manipulation from Abstract to Concrete](https://mlsanity.com/paper/2502.21257/) (2025)
- [Diffuse, Attend, and Segment: Unsupervised Zero-Shot Segmentation using Stable Diffusion](https://mlsanity.com/paper/2308.12469/) (2023)
- [Autoencoders as Cross-Modal Teachers: Can Pretrained 2D Image Transformers Help 3D Representation Learning?](https://mlsanity.com/paper/2212.08320/) (2022)
- [Affordance Diffusion: Synthesizing Hand-Object Interactions](https://mlsanity.com/paper/2303.12538/) (2023)
- [3D AffordanceNet: A Benchmark for Visual Object Affordance Understanding](https://mlsanity.com/paper/2103.16397/) (2021)
- [Grounding 3D Object Affordance from 2D Interactions in Images](https://mlsanity.com/paper/2303.10437/) (2023)
- [One-Shot Open Affordance Learning with Foundation Models](https://mlsanity.com/paper/2311.17776/) (2023)
- [Open-Vocabulary Affordance Detection in 3D Point Clouds](https://mlsanity.com/paper/2303.02401/) (2023)
- [Uni3D: A Unified Baseline for Multi-dataset 3D Object Detection](https://mlsanity.com/paper/2303.06880/) (2023)
- [Point-Cloud Completion with Pretrained Text-to-image Diffusion Models](https://mlsanity.com/paper/2306.10533/) (2023)
- [Cross-modal Learning for Image-Guided Point Cloud Shape Completion](https://mlsanity.com/paper/2209.09552/) (2022)
- [Beyond the Contact: Discovering Comprehensive Affordance for 3D Objects from Pre-trained 2D Diffusion Models](https://mlsanity.com/paper/2401.12978/) (2024)

## Built on by (most cited first, 6 of 6)

- [Affordance-R1: Reinforcement Learning for Generalizable Affordance Reasoning in Multimodal Large Language Model](https://mlsanity.com/paper/2508.06206/) (2025)
- [Affordance Agent Harness: Verification-Gated Skill Orchestration](https://mlsanity.com/paper/2605.00663/) (2026)
- [UniHM: Unified Dexterous Hand Manipulation with Vision Language Model](https://mlsanity.com/paper/2603.00732/) (2026)
- [HOID-R1: Reinforcement Learning for Open-World Human-Object Interaction Detection Reasoning with Multimodal Large Language Model](https://mlsanity.com/paper/2508.11350/) (2025)
- [VideoAfford: Grounding 3D Affordance from Human-Object-Interaction Videos via Multimodal Large Language Model](https://mlsanity.com/paper/2602.09638/) (2026)
- [PhotoHOI: Synthesizing 3D Hand-Object Interactions from a Single RGB Photograph](https://mlsanity.com/paper/2608.01905/) (2026)

## Abstract (arXiv)

3D affordance grounding aims to understand how diverse objects can be manipulated, making it a cornerstone of embodied interaction. However, prior works struggle to generalize to out-of-distribution, open-world scenarios, leaving a critical gap between limited dataset performance and real-world application needs. Inspired by the saying: textit``What I can not create, I do not understand'', we find generative models can generate semantically valid HOI images, which indicates inherent encoding of affordance concepts. Building on this insight, we propose DAG, the first innovative diffusion-based 3D affordance grounding framework that extracts general affordance knowledge from text-to-image diffusion models for 3D affordance prediction. Specifically, we extract the affordance priors from a diffusion model to encode HOI priors, and design an affordance block with a multi-source affordance decoder for dense 3D affordance prediction. Extensive experiments show that DAG consistently outperforms state-of-the-art methods and exhibits strong open-world generalization, even in the challenging one-shot setting. The code of our method is released on textcolorbluetextithttps://github.com/hq-King/DAG.