# Predicting Stable Configurations for Semantic Placement of Novel Objects

arXiv: https://arxiv.org/abs/2108.12062 · HTML: https://mlsanity.com/paper/2108.12062/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 40 references that are in the map.
- First posted on arXiv 26 Aug 2021.

## Builds on (most cited first, 23 of 23)

- [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017)
- [ShapeNet: An Information-Rich 3D Model Repository](https://mlsanity.com/paper/1512.03012/) (2015)
- [Integrated Task and Motion Planning](https://mlsanity.com/paper/2010.01083/) (2020)
- [6-DOF GraspNet: Variational Grasp Generation for Object Manipulation](https://mlsanity.com/paper/1905.10520/) (2019)
- [Contact-GraspNet: Efficient 6-DoF Grasp Generation in Cluttered Scenes](https://mlsanity.com/paper/2103.14127/) (2021)
- [Rearrangement: A Challenge for Embodied AI](https://mlsanity.com/paper/2011.01975/) (2020)
- [ACRONYM: A Large-Scale Grasp Dataset Based on Simulation](https://mlsanity.com/paper/2011.09584/) (2020)
- [Hierarchical Planning for Long-Horizon Manipulation with Geometric and Symbolic Scene Graphs](https://mlsanity.com/paper/2012.07277/) (2020)
- [Learning RGB-D Feature Embeddings for Unseen Object Instance Segmentation](https://mlsanity.com/paper/2007.15157/) (2020)
- [Neural Task Graphs: Generalizing to Unseen Tasks from a Single Video Demonstration](https://mlsanity.com/paper/1807.03480/) (2018)
- [Learning Physical Graph Representations from Visual Scenes](https://mlsanity.com/paper/2006.12373/) (2020)
- [Deep Affordance Foresight: Planning Through What Can Be Done in the Future](https://mlsanity.com/paper/2011.08424/) (2020)
- [NeRP: Neural Rearrangement Planning for Unknown Objects](https://mlsanity.com/paper/2106.01352/) (2021)
- [Regression Planning Networks](https://mlsanity.com/paper/1909.13072/) (2019)
- [A Billion Ways to Grasp: An Evaluation of Grasp Sampling Schemes on a Dense, Physics-based Grasp Data Set](https://mlsanity.com/paper/1912.05604/) (2019)
- [Transferable Task Execution from Pixels through Deep Planning Domain Learning](https://mlsanity.com/paper/2003.03726/) (2020)
- [Learning Object Placements For Relational Instructions by Hallucinating Scene Representations](https://mlsanity.com/paper/2001.08481/) (2020)
- [Spatial Reasoning from Natural Language Instructions for Robot Manipulation](https://mlsanity.com/paper/2012.13693/) (2020)
- [Visual Robot Task Planning](https://mlsanity.com/paper/1804.00062/) (2018)
- [Learning to Manipulate Object Collections Using Grounded State Representations](https://mlsanity.com/paper/1909.07876/) (2019)
- [Continuous Relaxation of Symbolic Planner for One-Shot Imitation Learning](https://mlsanity.com/paper/1908.06769/) (2019)
- [Relational Learning for Skill Preconditions](https://mlsanity.com/paper/2012.01693/) (2020)
- [Do What I Want, Not What I Did: Imitation of Skills by Planning Sequences of Actions](https://mlsanity.com/paper/1612.01215/) (2016)

## Built on by (most cited first, 24 of 24)

- [HomeRobot: Open-Vocabulary Mobile Manipulation](https://mlsanity.com/paper/2306.11565/) (2023)
- [StructFormer: Learning Spatial Structure for Language-Guided Semantic Rearrangement of Novel Objects](https://mlsanity.com/paper/2110.10189/) (2021)
- [Robot See Robot Do: Imitating Articulated Object Manipulation with Monocular 4D Reconstruction](https://mlsanity.com/paper/2409.18121/) (2024)
- [SE(3)-Equivariant Relational Rearrangement with Neural Descriptor Fields](https://mlsanity.com/paper/2211.09786/) (2022)
- [StructDiffusion: Language-Guided Creation of Physically-Valid Structures using Unseen Objects](https://mlsanity.com/paper/2211.04604/) (2022)
- [IFOR: Iterative Flow Minimization for Robotic Object Rearrangement](https://mlsanity.com/paper/2202.00732/) (2022)
- [Compositional Diffusion-Based Continuous Constraint Solvers](https://mlsanity.com/paper/2309.00966/) (2023)
- [Shelving, Stacking, Hanging: Relational Pose Diffusion for Multi-modal Rearrangement](https://mlsanity.com/paper/2307.04751/) (2023)
- [Composable Part-Based Manipulation](https://mlsanity.com/paper/2405.05876/) (2024)
- [Dream2Real: Zero-Shot 3D Object Rearrangement with Vision-Language Models](https://mlsanity.com/paper/2312.04533/) (2023)
- [Learning Perceptual Concepts by Bootstrapping from Human Queries](https://mlsanity.com/paper/2111.05251/) (2021)
- [Aligning Robot and Human Representations](https://mlsanity.com/paper/2302.01928/) (2023)
- [LINGO-Space: Language-Conditioned Incremental Grounding for Space](https://mlsanity.com/paper/2402.01183/) (2024)
- [Out of Sight, Still in Mind: Reasoning and Planning about Unobserved Objects with Video Tracking Enabled Memory Models](https://mlsanity.com/paper/2309.15278/) (2023)
- [Grasp, See, and Place: Efficient Unknown Object Rearrangement with Policy Structure Prior](https://mlsanity.com/paper/2402.15402/) (2024)
- [Spatial-Language Attention Policies for Efficient Robot Learning](https://mlsanity.com/paper/2304.11235/) (2023)
- [Slot-Level Robotic Placement via Visual Imitation from Single Human Video](https://mlsanity.com/paper/2504.01959/) (2025)
- [6-DoF Stability Field via Diffusion Models](https://mlsanity.com/paper/2310.17649/) (2023)
- [AnySlot: Goal-Conditioned Vision-Language-Action Policies for Zero-Shot Slot-Level Placement](https://mlsanity.com/paper/2604.10432/) (2026)
- [Simultaneous Pick and Place Detection by Combining SE(3) Diffusion Models with Differential Kinematics](https://mlsanity.com/paper/2504.19502/) (2025)
- [Robotic Test Tube Rearrangement Using Combined Reinforcement Learning and Motion Planning](https://mlsanity.com/paper/2401.09772/) (2024)
- [PACA: Perspective-Aware Cross-Attention Representation for Zero-Shot Scene Rearrangement](https://mlsanity.com/paper/2410.22059/) (2024)
- [Generating Stable Placements via Physics-guided Diffusion Models](https://mlsanity.com/paper/2509.21664/) (2025)
- [Explaining Why Things Go Where They Go: Interpretable Constructs of Human Organizational Preferences](https://mlsanity.com/paper/2512.24829/) (2025)

## Abstract (arXiv)

Human environments contain numerous objects configured in a variety of arrangements. Our goal is to enable robots to repose previously unseen objects according to learned semantic relationships in novel environments. We break this problem down into two parts: (1) finding physically valid locations for the objects and (2) determining if those poses satisfy learned, high-level semantic relationships. We build our models and training from the ground up to be tightly integrated with our proposed planning algorithm for semantic placement of unknown objects. We train our models purely in simulation, with no fine-tuning needed for use in the real world. Our approach enables motion planning for semantic rearrangement of unknown objects in scenes with varying geometry from only RGB-D sensing. Our experiments through a set of simulated ablations demonstrate that using a relational classifier alone is not sufficient for reliable planning. We further demonstrate the ability of our planner to generate and execute diverse manipulation plans through a set of real-world experiments with a variety of objects.