# Attribute-based Object Grounding and Robot Grasp Detection with Spatial Reasoning

arXiv: https://arxiv.org/abs/2509.08126 · HTML: https://mlsanity.com/paper/2509.08126/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Spatial and embodied reasoning, one of 17 directions in LLM reasoning.
- Builds on 35 of its 39 references that are in the map.
- First posted on arXiv 9 Sep 2025.

## Placed in

- LLM reasoning / [Spatial and embodied reasoning](https://mlsanity.com/lr/spatial-and-embodied-reasoning/)

## Builds on (most cited first, 30 of 35)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://mlsanity.com/paper/2201.12086/) (2022)
- [OpenVLA: An Open-Source Vision-Language-Action Model](https://mlsanity.com/paper/2406.09246/) (2024)
- [FiLM: Visual Reasoning with a General Conditioning Layer](https://mlsanity.com/paper/1709.07871/) (2017)
- [CLIPort: What and Where Pathways for Robotic Manipulation](https://mlsanity.com/paper/2109.12098/) (2021)
- [VIMA: General Robot Manipulation with Multimodal Prompts](https://mlsanity.com/paper/2210.03094/) (2022)
- [CRIS: CLIP-Driven Referring Image Segmentation](https://mlsanity.com/paper/2111.15174/) (2021)
- [LAVT: Language-Aware Vision Transformer for Referring Image Segmentation](https://mlsanity.com/paper/2112.02244/) (2021)
- [Robotic Control via Embodied Chain-of-Thought Reasoning](https://mlsanity.com/paper/2407.08693/) (2024)
- [Segmentation from Natural Language Expressions](https://mlsanity.com/paper/1603.06180/) (2016)
- [Recurrent Multimodal Interaction for Referring Image Segmentation](https://mlsanity.com/paper/1703.07939/) (2017)
- [Grounding Language with Visual Affordances over Unstructured Data](https://mlsanity.com/paper/2210.01911/) (2022)
- [Beyond One-to-One: Rethinking the Referring Image Segmentation](https://mlsanity.com/paper/2308.13853/) (2023)
- [Bridging Vision and Language Encoders: Parameter-Efficient Tuning for Referring Image Segmentation](https://mlsanity.com/paper/2307.11545/) (2023)
- [Antipodal Robotic Grasping using Generative Residual Convolutional Neural Network](https://mlsanity.com/paper/1909.04810/) (2019)
- [Contrastive Grouping with Transformer for Referring Image Segmentation](https://mlsanity.com/paper/2309.01017/) (2023)
- [VL-Grasp: a 6-Dof Interactive Grasp Policy for Language-Oriented Objects in Cluttered Indoor Scenes](https://mlsanity.com/paper/2308.00640/) (2023)
- [A Joint Modeling of Vision-Language-Action for Target-oriented Grasping in Clutter](https://mlsanity.com/paper/2302.12610/) (2023)
- [Language-driven Grasp Detection](https://mlsanity.com/paper/2406.09489/) (2024)
- [Language-guided Robot Grasping: CLIP-based Referring Grasp Synthesis in Clutter](https://mlsanity.com/paper/2311.05779/) (2023)
- [Reasoning Grasping via Multimodal Large Language Model](https://mlsanity.com/paper/2402.06798/) (2024)
- [ThinkGrasp: A Vision-Language System for Strategic Part Grasping in Clutter](https://mlsanity.com/paper/2407.11298/) (2024)
- [Towards Open-World Grasping with Large Vision-Language Models](https://mlsanity.com/paper/2406.18722/) (2024)
- [GLOVER: Generalizable Open-Vocabulary Affordance Reasoning for Task-Oriented Grasping](https://mlsanity.com/paper/2411.12286/) (2024)
- [Self-Supervised Interactive Object Segmentation Through a Singulation-and-Grasping Approach](https://mlsanity.com/paper/2207.09314/) (2022)

## Abstract (arXiv)

Enabling robots to grasp objects specified through natural language is essential for effective human-robot interaction, yet it remains a significant challenge. Existing approaches often struggle with open-form language expressions and typically assume unambiguous target objects without duplicates. Moreover, they frequently rely on costly, dense pixel-wise annotations for both object grounding and grasp configuration. We present Attribute-based Object Grounding and Robotic Grasping (OGRG), a novel framework that interprets open-form language expressions and performs spatial reasoning to ground target objects and predict planar grasp poses, even in scenes containing duplicated object instances. We investigate OGRG in two settings: (1) Referring Grasp Synthesis (RGS) under pixel-wise full supervision, and (2) Referring Grasp Affordance (RGA) using weakly supervised learning with only single-pixel grasp annotations. Key contributions include a bi-directional vision-language fusion module and the integration of depth information to enhance geometric reasoning, improving both grounding and grasping performance. Experiment results show that OGRG outperforms strong baselines in tabletop scenes with diverse spatial language instructions. In RGS, it operates at 17.59 FPS on a single NVIDIA RTX 2080 Ti GPU, enabling potential use in closed-loop or multi-object sequential grasping, while delivering superior grounding and grasp prediction accuracy compared to all the baselines considered. Under the weakly supervised RGA setting, OGRG also surpasses baseline grasp-success rates in both simulation and real-robot trials, underscoring the effectiveness of its spatial reasoning design. Project page: https://z.umn.edu/ogrg