# RAGNet: Large-scale Reasoning-based Affordance Segmentation Benchmark towards General Grasping

arXiv: https://arxiv.org/abs/2507.23734 · HTML: https://mlsanity.com/paper/2507.23734/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 53 of its 69 references that are in the map.
- First posted on arXiv 31 Jul 2025.

## Builds on (most cited first, 30 of 53)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection](https://mlsanity.com/paper/2303.05499/) (2023)
- [SAM 2: Segment Anything in Images and Videos](https://mlsanity.com/paper/2408.00714/) (2024)
- [RT-1: Robotics Transformer for Real-World Control at Scale](https://mlsanity.com/paper/2212.06817/) (2022)
- [Ego4D: Around the World in 3,000 Hours of Egocentric Video](https://mlsanity.com/paper/2110.07058/) (2021)
- [Modeling Context in Referring Expressions](https://mlsanity.com/paper/1608.00272/) (2016)
- [COCO-Stuff: Thing and Stuff Classes in Context](https://mlsanity.com/paper/1612.03716/) (2016)
- [LISA: Reasoning Segmentation via Large Language Model](https://mlsanity.com/paper/2308.00692/) (2023)
- [VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Models](https://mlsanity.com/paper/2307.05973/) (2023)
- [RLBench: The Robot Learning Benchmark & Learning Environment](https://mlsanity.com/paper/1909.12271/) (2019)
- [BridgeData V2: A Dataset for Robot Learning at Scale](https://mlsanity.com/paper/2308.12952/) (2023)
- [Taskonomy: Disentangling Task Transfer Learning](https://mlsanity.com/paper/1804.08328/) (2018)
- [CLIPort: What and Where Pathways for Robotic Manipulation](https://mlsanity.com/paper/2109.12098/) (2021)
- [Rescaling Egocentric Vision](https://mlsanity.com/paper/2006.13256/) (2020)
- [GLaMM: Pixel Grounding Large Multimodal Model](https://mlsanity.com/paper/2311.03356/) (2023)
- [Florence-2: Advancing a Unified Representation for a Variety of Vision Tasks](https://mlsanity.com/paper/2311.06242/) (2023)
- [ReKep: Spatio-Temporal Reasoning of Relational Keypoint Constraints for Robotic Manipulation](https://mlsanity.com/paper/2409.01652/) (2024)
- [Understanding Human Hands in Contact at Internet Scale](https://mlsanity.com/paper/2006.06669/) (2020)
- [Affordances from Human Videos as a Versatile Representation for Robotics](https://mlsanity.com/paper/2304.08488/) (2023)
- [PIVOT: Iterative Visual Prompting Elicits Actionable Knowledge for VLMs](https://mlsanity.com/paper/2402.07872/) (2024)
- [HOTR: End-to-End Human-Object Interaction Detection with Transformers](https://mlsanity.com/paper/2104.13682/) (2021)
- [ManipLLM: Embodied Multimodal Large Language Model for Object-Centric Robotic Manipulation](https://mlsanity.com/paper/2312.16217/) (2023)
- [PACO: Parts and Attributes of Common Objects](https://mlsanity.com/paper/2301.01795/) (2023)
- [AffordanceNet: An End-to-End Deep Learning Approach for Object Affordance Detection](https://mlsanity.com/paper/1709.07326/) (2017)
- [CoPa: General Robotic Manipulation through Spatial Constraints of Parts with Foundation Models](https://mlsanity.com/paper/2403.08248/) (2024)

## Built on by (most cited first, 11 of 11)

- [MemoryVLA: Perceptual-Cognitive Memory in Vision-Language-Action Models for Robotic Manipulation](https://mlsanity.com/paper/2508.19236/) (2025)
- [SpatialActor: Exploring Disentangled Spatial Representations for Robust Robotic Manipulation](https://mlsanity.com/paper/2511.09555/) (2025)
- [From Web to Pixels: Bringing Agentic Search into Visual Perception](https://mlsanity.com/paper/2605.12497/) (2026)
- [AffordanceGrasp-R1:Leveraging Reasoning-Based Affordance Segmentation with Reinforcement Learning for Robotic Grasping](https://mlsanity.com/paper/2602.03547/) (2026)
- [Panoramic Affordance Prediction](https://mlsanity.com/paper/2603.15558/) (2026)
- [Learning Visual Affordance from Audio](https://mlsanity.com/paper/2512.02005/) (2025)
- [Uncertainty Estimation in Instance Segmentation of Affordances via Bayesian Visual Transformers](https://mlsanity.com/paper/2605.03614/) (2026)
- [AFUN: Towards an Affordance Foundation Model for Functionality Understanding](https://mlsanity.com/paper/2606.02551/) (2026)
- [EgoAfford: Task-Oriented Affordance Grounding via Egocentric Referring Segmentation](https://mlsanity.com/paper/2608.04533/) (2026)
- [Zero-shot 2D Grounding with Novel Affordance Types](https://mlsanity.com/paper/2608.08929/) (2026)
- [Are Frontier VLM Agents Ready to Be Robot Generalists? An Empirical Study with the Embodied Agent Arena](https://mlsanity.com/paper/2610.00854/) (2026)

## Abstract (arXiv)

General robotic grasping systems require accurate object affordance perception in diverse open-world scenarios following human instructions. However, current studies suffer from the problem of lacking reasoning-based large-scale affordance prediction data, leading to considerable concern about open-world effectiveness. To address this limitation, we build a large-scale grasping-oriented affordance segmentation benchmark with human-like instructions, named RAGNet. It contains 273k images, 180 categories, and 26k reasoning instructions. The images cover diverse embodied data domains, such as wild, robot, ego-centric, and even simulation data. They are carefully annotated with an affordance map, while the difficulty of language instructions is largely increased by removing their category name and only providing functional descriptions. Furthermore, we propose a comprehensive affordance-based grasping framework, named AffordanceNet, which consists of a VLM pre-trained on our massive affordance data and a grasping network that conditions an affordance map to grasp the target. Extensive experiments on affordance segmentation benchmarks and real-robot manipulation tasks show that our model has a powerful open-world generalization ability. Our data and code is available at https://github.com/wudongming97/AffordanceNet.