# Learning Object Placements For Relational Instructions by Hallucinating Scene Representations

arXiv: https://arxiv.org/abs/2001.08481 · HTML: https://mlsanity.com/paper/2001.08481/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Jan 2020.

## Built on by (most cited first, 16 of 16)

- [Benchmarking Spatial Relationships in Text-to-Image Generation](https://mlsanity.com/paper/2212.10015/) (2022)
- [StructFormer: Learning Spatial Structure for Language-Guided Semantic Rearrangement of Novel Objects](https://mlsanity.com/paper/2110.10189/) (2021)
- [Affordance Learning from Play for Sample-Efficient Policy Learning](https://mlsanity.com/paper/2203.00352/) (2022)
- [Predicting Stable Configurations for Semantic Placement of Novel Objects](https://mlsanity.com/paper/2108.12062/) (2021)
- [InsActor: Instruction-driven Physics-based Characters](https://mlsanity.com/paper/2312.17135/) (2023)
- [Composing Pick-and-Place Tasks By Grounding Language](https://mlsanity.com/paper/2102.08094/) (2021)
- [LINGO-Space: Language-Conditioned Incremental Grounding for Space](https://mlsanity.com/paper/2402.01183/) (2024)
- [Differentiable Parsing and Visual Grounding of Natural Language Instructions for Object Placement](https://mlsanity.com/paper/2210.00215/) (2022)
- [Learning 3D Object Spatial Relationships from Pre-trained 2D Diffusion Models](https://mlsanity.com/paper/2503.19914/) (2025)
- [Object-centric Inference for Language Conditioned Placement: A Foundation Model based Approach](https://mlsanity.com/paper/2304.02893/) (2023)
- [GVCCI: Lifelong Learning of Visual Grounding for Language-Guided Robotic Manipulation](https://mlsanity.com/paper/2307.05963/) (2023)
- [Natural Language Instructions for Intuitive Human Interaction with Robotic Assistants in Field Construction Work](https://mlsanity.com/paper/2307.04195/) (2023)
- [Slot-Level Robotic Placement via Visual Imitation from Single Human Video](https://mlsanity.com/paper/2504.01959/) (2025)
- [DASH: Modularized Human Manipulation Simulation with Vision and Language for Embodied AI](https://mlsanity.com/paper/2108.12536/) (2021)
- [PGA: Personalizing Grasping Agents with Single Human-Robot Interaction](https://mlsanity.com/paper/2310.12547/) (2023)
- [Stimulating Imagination: Towards General-purpose "Something Something Placement"](https://mlsanity.com/paper/2408.01655/) (2024)

## Abstract (arXiv)

Robots coexisting with humans in their environment and performing services for them need the ability to interact with them. One particular requirement for such robots is that they are able to understand spatial relations and can place objects in accordance with the spatial relations expressed by their user. In this work, we present a convolutional neural network for estimating pixelwise object placement probabilities for a set of spatial relations from a single input image. During training, our network receives the learning signal by classifying hallucinated high-level scene representations as an auxiliary task. Unlike previous approaches, our method does not require ground truth data for the pixelwise relational probabilities or 3D models of the objects, which significantly expands the applicability in practical applications. Our results obtained using real-world data and human-robot experiments demonstrate the effectiveness of our method in reasoning about the best way to place objects to reproduce a spatial relation. Videos of our experiments can be found at https://youtu.be/zaZkHTWFMKM