# AFUN: Towards an Affordance Foundation Model for Functionality Understanding

arXiv: https://arxiv.org/abs/2606.02551 · HTML: https://mlsanity.com/paper/2606.02551/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 79 of its 101 references that are in the map.
- First posted on arXiv 1 Jun 2026.

## Builds on (most cited first, 30 of 79)

- [Qwen3-VL Technical Report](https://mlsanity.com/paper/2511.21631/) (2025)
- [Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations](https://mlsanity.com/paper/1602.07332/) (2016)
- [Ego4D: Around the World in 3,000 Hours of Egocentric Video](https://mlsanity.com/paper/2110.07058/) (2021)
- [SAM 3: Segment Anything with Concepts](https://mlsanity.com/paper/2511.16719/) (2025)
- [DROID: A Large-Scale In-The-Wild Robot Manipulation Dataset](https://mlsanity.com/paper/2403.12945/) (2024)
- [LISA: Reasoning Segmentation via Large Language Model](https://mlsanity.com/paper/2308.00692/) (2023)
- [Depth Anything 3: Recovering the Visual Space from Any Views](https://mlsanity.com/paper/2511.10647/) (2025)
- [RLBench: The Robot Learning Benchmark & Learning Environment](https://mlsanity.com/paper/1909.12271/) (2019)
- [CALVIN: A Benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Tasks](https://mlsanity.com/paper/2112.03227/) (2021)
- [Rescaling Egocentric Vision](https://mlsanity.com/paper/2006.13256/) (2020)
- [AgiBot World Colosseo: A Large-scale Manipulation Platform for Scalable and Intelligent Embodied Systems](https://mlsanity.com/paper/2503.06669/) (2025)
- [RoboPoint: A Vision-Language Model for Spatial Affordance Prediction for Robotics](https://mlsanity.com/paper/2406.10721/) (2024)
- [Transfer between Modalities with MetaQueries](https://mlsanity.com/paper/2504.06256/) (2025)
- [Affordances from Human Videos as a Versatile Representation for Robotics](https://mlsanity.com/paper/2304.08488/) (2023)
- [AnyGrasp: Robust and Efficient Grasp Perception in Spatial and Temporal Domains](https://mlsanity.com/paper/2212.08333/) (2022)
- [RH20T: A Comprehensive Robotic Dataset for Learning Diverse Skills in One-Shot](https://mlsanity.com/paper/2307.00595/) (2023)
- [Where2Act: From Pixels to Actions for Articulated 3D Objects](https://mlsanity.com/paper/2101.02692/) (2021)
- [RoboMIND: Benchmark on Multi-embodiment Intelligence Normative Data for Robot Manipulation](https://mlsanity.com/paper/2412.13877/) (2024)
- [ManipLLM: Embodied Multimodal Large Language Model for Object-Centric Robotic Manipulation](https://mlsanity.com/paper/2312.16217/) (2023)
- [HOT3D: Hand and Object Tracking in 3D from Egocentric Multi-View Videos](https://mlsanity.com/paper/2411.19167/) (2024)
- [AffordanceNet: An End-to-End Deep Learning Approach for Object Affordance Detection](https://mlsanity.com/paper/1709.07326/) (2017)
- [Track2Act: Predicting Point Tracks from Internet Videos enables Generalizable Robot Manipulation](https://mlsanity.com/paper/2405.01527/) (2024)
- [Joint Hand Motion and Interaction Hotspots Prediction from Egocentric Videos](https://mlsanity.com/paper/2204.01696/) (2022)
- [Sonata: Self-Supervised Learning of Reliable Point Representations](https://mlsanity.com/paper/2503.16429/) (2025)
- [3D AffordanceNet: A Benchmark for Visual Object Affordance Understanding](https://mlsanity.com/paper/2103.16397/) (2021)
- [Fast Interactive Object Annotation with Curve-GCN](https://mlsanity.com/paper/1903.06874/) (2019)
- [FlowBot3D: Learning 3D Articulation Flow to Manipulate Articulated Objects](https://mlsanity.com/paper/2205.04382/) (2022)
- [General Flow as Foundation Affordance for Scalable Robot Learning](https://mlsanity.com/paper/2401.11439/) (2024)
- [Learning Affordance Grounding from Exocentric Images](https://mlsanity.com/paper/2203.09905/) (2022)
- [VAT-Mart: Learning Visual Action Trajectory Proposals for Manipulating 3D ARTiculated Objects](https://mlsanity.com/paper/2106.14440/) (2021)

## Abstract (arXiv)

Affordance understanding bridges visual perception and physical action, serving as an explainable interface for robot manipulation in open and unstructured real-world environments. Yet, building an affordance foundation model that not only understands where and how the interaction should happen, but also generalizes across diverse environments, objects, and tasks, remains a long-standing research challenge. Existing methods typically address only part of this challenge, either localizing task-relevant regions without specifying executable motion, or predicting motion but with limited scalability. In this paper, we present ourmodel, a step towards an affordance foundation model for functionality understanding. From a single RGB-D observation and a language task description, ourmodel predicts a task-conditional functional mask (where to interact) and a 3D post-contact motion curve (how to interact). To support open-world generalization, we build a large-scale standardized data pipeline that converts heterogeneous robot, human, simulation, and real-world scan data into a shared affordance schema with language, masks, and object-centric 3D motion labels. We evaluate ourmodel from three aspects: for affordance segmentation, ourmodel outperforms all baselines by a large margin across 8 test sets from 4 benchmarks, improving mean gIoU/cIoU by +23.9/+26.3; for contact-point prediction, it predicts substantially more accurate points, with a 12.7--61.3% hit-rate gain over the best baseline; and for 3D motion, it achieves the best performance on all three test sets. ourmodel can be deployed for real-world robot manipulation without finetuning for robot embodiment or using task-specific heuristics, demonstrating the ability to adapt to open-world affordance tasks. Project page: https://www.zhaoningwang.com/AFUN