# Transferable Task Execution from Pixels through Deep Planning Domain Learning

arXiv: https://arxiv.org/abs/2003.03726 · HTML: https://mlsanity.com/paper/2003.03726/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Mar 2020.

## Built on by (most cited first, 17 of 17)

- [StructFormer: Learning Spatial Structure for Language-Guided Semantic Rearrangement of Novel Objects](https://mlsanity.com/paper/2110.10189/) (2021)
- [Reasoning Paths with Reference Objects Elicit Quantitative Spatial Reasoning in Large Vision-Language Models](https://mlsanity.com/paper/2409.09788/) (2024)
- [SORNet: Spatial Object-Centric Representations for Sequential Manipulation](https://mlsanity.com/paper/2109.03891/) (2021)
- [Adversarial Skill Chaining for Long-Horizon Robot Manipulation via Terminal State Regularization](https://mlsanity.com/paper/2111.07999/) (2021)
- [Predicting Stable Configurations for Semantic Placement of Novel Objects](https://mlsanity.com/paper/2108.12062/) (2021)
- [Guided Imitation of Task and Motion Planning](https://mlsanity.com/paper/2112.03386/) (2021)
- [Composable Part-Based Manipulation](https://mlsanity.com/paper/2405.05876/) (2024)
- [Learning Perceptual Concepts by Bootstrapping from Human Queries](https://mlsanity.com/paper/2111.05251/) (2021)
- [Learning to Search in Task and Motion Planning with Streams](https://mlsanity.com/paper/2111.13144/) (2021)
- [NOD-TAMP: Generalizable Long-Horizon Planning with Neural Object Descriptors](https://mlsanity.com/paper/2311.01530/) (2023)
- [Task and Motion Planning for Execution in the Real](https://mlsanity.com/paper/2406.03641/) (2024)
- [Evaluating Continual Learning on a Home Robot](https://mlsanity.com/paper/2306.02413/) (2023)
- [Online Grounding of Symbolic Planning Domains in Unknown Environments](https://mlsanity.com/paper/2112.10007/) (2021)
- [Cloud-Based Hierarchical Imitation Learning for Scalable Transfer of Construction Skills from Human Workers to Assisting Robots](https://mlsanity.com/paper/2309.11619/) (2023)
- [Learning Symbolic Operators: A Neurosymbolic Solution for Autonomous Disassembly of Electric Vehicle Battery](https://mlsanity.com/paper/2206.03027/) (2022)
- [Policy-Guided Lazy Search with Feedback for Task and Motion Planning](https://mlsanity.com/paper/2210.14055/) (2022)
- [Learning Top-k Subtask Planning Tree based on Discriminative Representation Pre-training for Decision Making](https://mlsanity.com/paper/2312.11027/) (2023)

## Abstract (arXiv)

While robots can learn models to solve many manipulation tasks from raw visual input, they cannot usually use these models to solve new problems. On the other hand, symbolic planning methods such as STRIPS have long been able to solve new problems given only a domain definition and a symbolic goal, but these approaches often struggle on the real world robotic tasks due to the challenges of grounding these symbols from sensor data in a partially-observable world. We propose Deep Planning Domain Learning (DPDL), an approach that combines the strengths of both methods to learn a hierarchical model. DPDL learns a high-level model which predicts values for a large set of logical predicates consisting of the current symbolic world state, and separately learns a low-level policy which translates symbolic operators into executable actions on the robot. This allows us to perform complex, multi-step tasks even when the robot has not been explicitly trained on them. We show our method on manipulation tasks in a photorealistic kitchen scenario.