# Structural Concept Learning via Graph Attention for Multi-Level Rearrangement Planning

arXiv: https://arxiv.org/abs/2309.02547 · HTML: https://mlsanity.com/paper/2309.02547/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 40 references that are in the map.
- First posted on arXiv 5 Sep 2023.

## Builds on (most cited first, 30 of 31)

- [Graph Attention Networks](https://mlsanity.com/paper/1710.10903/) (2017)
- [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017)
- [Fast Graph Representation Learning with PyTorch Geometric](https://mlsanity.com/paper/1903.02428/) (2019)
- [How Attentive are Graph Attention Networks?](https://mlsanity.com/paper/2105.14491/) (2021)
- [Transporter Networks: Rearranging the Visual World for Robotic Manipulation](https://mlsanity.com/paper/2010.14406/) (2020)
- [Integrated Task and Motion Planning](https://mlsanity.com/paper/2010.01083/) (2020)
- [TEASER: Fast and Certifiable Point Cloud Registration](https://mlsanity.com/paper/2001.07715/) (2020)
- [Rearrangement: A Challenge for Embodied AI](https://mlsanity.com/paper/2011.01975/) (2020)
- [PDDLStream: Integrating Symbolic Planners and Blackbox Samplers via Optimistic Adaptive Planning](https://mlsanity.com/paper/1802.08705/) (2018)
- [iSDF: Real-Time Neural Signed Distance Fields for Robot Perception](https://mlsanity.com/paper/2204.02296/) (2022)
- [DALL-E-Bot: Introducing Web-Scale Diffusion Models to Robotics](https://mlsanity.com/paper/2210.02438/) (2022)
- [Visual Room Rearrangement](https://mlsanity.com/paper/2103.16544/) (2021)
- [Hierarchical Planning for Long-Horizon Manipulation with Geometric and Symbolic Scene Graphs](https://mlsanity.com/paper/2012.07277/) (2020)
- [LEGO-Net: Learning Regular Rearrangements of Objects in Rooms](https://mlsanity.com/paper/2301.09629/) (2023)
- [StructFormer: Learning Spatial Structure for Language-Guided Semantic Rearrangement of Novel Objects](https://mlsanity.com/paper/2110.10189/) (2021)
- [Object Rearrangement Using Learned Implicit Collision Functions](https://mlsanity.com/paper/2011.10726/) (2020)
- [IFOR: Iterative Flow Minimization for Robotic Object Rearrangement](https://mlsanity.com/paper/2202.00732/) (2022)
- [NeRP: Neural Rearrangement Planning for Unknown Objects](https://mlsanity.com/paper/2106.01352/) (2021)
- [Monte-Carlo Tree Search for Efficient Visually Guided Rearrangement Planning](https://mlsanity.com/paper/1904.10348/) (2019)
- [Long-Horizon Multi-Robot Rearrangement Planning for Construction Assembly](https://mlsanity.com/paper/2106.02489/) (2021)
- [Blocks Assemble! Learning to Assemble with Large-Scale Structured Reinforcement Learning](https://mlsanity.com/paper/2203.13733/) (2022)
- [Semantically Grounded Object Matching for Robust Robotic Scene Rearrangement](https://mlsanity.com/paper/2111.07975/) (2021)
- [CabiNet: Scaling Neural Collision Detection for Object Rearrangement with Procedural Scene Generation](https://mlsanity.com/paper/2304.09302/) (2023)
- [Where to relocate?: Object rearrangement inside cluttered and confined environments for robotic manipulation](https://mlsanity.com/paper/2003.10863/) (2020)
- [RoboAssembly: Learning Generalizable Furniture Assembly Policy in a Novel Multi-robot Contact-rich Simulation Environment](https://mlsanity.com/paper/2112.10143/) (2021)
- [Multi-Object Rearrangement with Monte Carlo Tree Search:A Case Study on Planar Nonprehensile Sorting](https://mlsanity.com/paper/1912.07024/) (2019)
- [TarGF: Learning Target Gradient Field to Rearrange Objects without Explicit Goal Specification](https://mlsanity.com/paper/2209.00853/) (2022)
- [Planning for Multi-Object Manipulation with Graph Neural Network Relational Classifiers](https://mlsanity.com/paper/2209.11943/) (2022)
- [Uniform Object Rearrangement: From Complete Monotone Primitives to Efficient Non-Monotone Informed Search](https://mlsanity.com/paper/2101.12241/) (2021)
- [Fast High-Quality Tabletop Rearrangement in Bounded Workspace](https://mlsanity.com/paper/2110.12325/) (2021)

## Built on by (most cited first, 8 of 8)

- [InCoRo: In-Context Learning for Robotics Control with Feedback Loops](https://mlsanity.com/paper/2402.05188/) (2024)
- [Out of Sight, Still in Mind: Reasoning and Planning about Unobserved Objects with Video Tracking Enabled Memory Models](https://mlsanity.com/paper/2309.15278/) (2023)
- [Multi-Object Graph Affordance Network: Goal-Oriented Planning through Learned Compound Object Affordances](https://mlsanity.com/paper/2309.10426/) (2023)
- [Learning to Build: Autonomous Robotic Assembly of Stable Structures Without Predefined Plans](https://mlsanity.com/paper/2602.23934/) (2026)
- [Robotic Test Tube Rearrangement Using Combined Reinforcement Learning and Motion Planning](https://mlsanity.com/paper/2401.09772/) (2024)
- [PACA: Perspective-Aware Cross-Attention Representation for Zero-Shot Scene Rearrangement](https://mlsanity.com/paper/2410.22059/) (2024)
- [VLAD-Grasp: Zero-shot Grasp Detection via Vision-Language Models](https://mlsanity.com/paper/2511.05791/) (2025)
- [EUPHORIA: Efficient Universal Planning via Hybrid Optimization for Robust Industrial Robotic Assembly](https://mlsanity.com/paper/2605.18872/) (2026)

## Abstract (arXiv)

Robotic manipulation tasks, such as object rearrangement, play a crucial role in enabling robots to interact with complex and arbitrary environments. Existing work focuses primarily on single-level rearrangement planning and, even if multiple levels exist, dependency relations among substructures are geometrically simpler, like tower stacking. We propose Structural Concept Learning (SCL), a deep learning approach that leverages graph attention networks to perform multi-level object rearrangement planning for scenes with structural dependency hierarchies. It is trained on a self-generated simulation data set with intuitive structures, works for unseen scenes with an arbitrary number of objects and higher complexity of structures, infers independent substructures to allow for task parallelization over multiple manipulators, and generalizes to the real world. We compare our method with a range of classical and model-based baselines to show that our method leverages its scene understanding to achieve better performance, flexibility, and efficiency. The dataset, supplementary details, videos, and code implementation are available at: https://manavkulshrestha.github.io/scl