# Spatial Reasoning from Natural Language Instructions for Robot Manipulation

arXiv: https://arxiv.org/abs/2012.13693 · HTML: https://mlsanity.com/paper/2012.13693/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 Dec 2020.

## Built on by (most cited first, 16 of 16)

- [CALVIN: A Benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Tasks](https://mlsanity.com/paper/2112.03227/) (2021)
- [What Matters in Language Conditioned Robotic Imitation Learning over Unstructured Data](https://mlsanity.com/paper/2204.06252/) (2022)
- [StructFormer: Learning Spatial Structure for Language-Guided Semantic Rearrangement of Novel Objects](https://mlsanity.com/paper/2110.10189/) (2021)
- [Transfer Learning with Synthetic Corpora for Spatial Role Labeling and Reasoning](https://mlsanity.com/paper/2210.16952/) (2022)
- [UGIF: UI Grounded Instruction Following](https://mlsanity.com/paper/2211.07615/) (2022)
- [Predicting Stable Configurations for Semantic Placement of Novel Objects](https://mlsanity.com/paper/2108.12062/) (2021)
- [CoPESD: A Multi-Level Surgical Motion Dataset for Training Large Vision-Language Models to Co-Pilot Endoscopic Submucosal Dissection](https://mlsanity.com/paper/2410.07540/) (2024)
- [LINGO-Space: Language-Conditioned Incremental Grounding for Space](https://mlsanity.com/paper/2402.01183/) (2024)
- [ViCLEVR: A Visual Reasoning Dataset and Hybrid Multimodal Fusion Model for Visual Question Answering in Vietnamese](https://mlsanity.com/paper/2310.18046/) (2023)
- [SpaRC and SpaRP: Spatial Reasoning Characterization and Path Generation for Understanding Spatial Reasoning Capability of Large Language Models](https://mlsanity.com/paper/2406.04566/) (2024)
- [GVCCI: Lifelong Learning of Visual Grounding for Language-Guided Robotic Manipulation](https://mlsanity.com/paper/2307.05963/) (2023)
- [DepWiGNN: A Depth-wise Graph Neural Network for Multi-hop Spatial Reasoning in Text](https://mlsanity.com/paper/2310.12557/) (2023)
- [PROGrasp: Pragmatic Human-Robot Communication for Object Grasping](https://mlsanity.com/paper/2309.07759/) (2023)
- [PGA: Personalizing Grasping Agents with Single Human-Robot Interaction](https://mlsanity.com/paper/2310.12547/) (2023)
- [Stimulating Imagination: Towards General-purpose "Something Something Placement"](https://mlsanity.com/paper/2408.01655/) (2024)
- [C2F-Space: Coarse-to-Fine Space Grounding for Spatial Instructions using Vision-Language Models](https://mlsanity.com/paper/2511.15333/) (2025)

## Abstract (arXiv)

Robots that can manipulate objects in unstructured environments and collaborate with humans can benefit immensely by understanding natural language. We propose a pipelined architecture of two stages to perform spatial reasoning on the text input. All the objects in the scene are first localized, and then the instruction for the robot in natural language and the localized co-ordinates are mapped to the start and end co-ordinates corresponding to the locations where the robot must pick up and place the object respectively. We show that representing the localized objects by quantizing their positions to a binary grid is preferable to representing them as a list of 2D co-ordinates. We also show that attention improves generalization and can overcome biases in the dataset. The proposed method is used to pick-and-place playing cards using a robot arm.