# Generalizable Task Planning through Representation Pretraining

arXiv: https://arxiv.org/abs/2205.07993 · HTML: https://mlsanity.com/paper/2205.07993/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 May 2022.

## Built on by (most cited first, 19 of 19)

- [VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Models](https://mlsanity.com/paper/2307.05973/) (2023)
- [Multi-Stage Cable Routing through Hierarchical Imitation Learning](https://mlsanity.com/paper/2307.08927/) (2023)
- [Learning Neuro-Symbolic Skills for Bilevel Planning](https://mlsanity.com/paper/2206.10680/) (2022)
- [Sequential Dexterity: Chaining Dexterous Policies for Long-Horizon Manipulation](https://mlsanity.com/paper/2309.00987/) (2023)
- [LEAGUE: Guided Skill Learning and Abstraction for Long-Horizon Manipulation](https://mlsanity.com/paper/2210.12631/) (2022)
- [A Survey of Optimization-based Task and Motion Planning: From Classical To Learning Approaches](https://mlsanity.com/paper/2404.02817/) (2024)
- [STAP: Sequencing Task-Agnostic Policies](https://mlsanity.com/paper/2210.12250/) (2022)
- [DANLI: Deliberative Agent for Following Natural Language Instructions](https://mlsanity.com/paper/2210.12485/) (2022)
- [Primitive Skill-based Robot Learning from Human Evaluative Feedback](https://mlsanity.com/paper/2307.15801/) (2023)
- [Can Transformers Capture Spatial Relations between Objects?](https://mlsanity.com/paper/2403.00729/) (2024)
- [Embodied Active Learning of Relational State Abstractions for Bilevel Planning](https://mlsanity.com/paper/2303.04912/) (2023)
- [NOIR: Neural Signal Operated Intelligent Robots for Everyday Activities](https://mlsanity.com/paper/2311.01454/) (2023)
- [NOD-TAMP: Generalizable Long-Horizon Planning with Neural Object Descriptors](https://mlsanity.com/paper/2311.01530/) (2023)
- [Learning Planning Abstractions from Language](https://mlsanity.com/paper/2405.03864/) (2024)
- [Recognising Affordances in Predicted Futures to Plan with Consideration of Non-canonical Affordance Effects](https://mlsanity.com/paper/2206.10920/) (2022)
- [Compositional Servoing by Recombining Demonstrations](https://mlsanity.com/paper/2310.04271/) (2023)
- [Knolling Bot: Teaching Robots the Human Notion of Tidiness](https://mlsanity.com/paper/2310.04566/) (2023)
- [NOIR 2.0: Neural Signal Operated Intelligent Robots for Everyday Activities](https://mlsanity.com/paper/2511.20848/) (2025)
- [Affordance-Based Manipulation Planning with Text Goals and Sim-to-Real Generalisation via Real-to-Sim Image Conversion](https://mlsanity.com/paper/2607.11004/) (2026)

## Abstract (arXiv)

The ability to plan for multi-step manipulation tasks in unseen situations is crucial for future home robots. But collecting sufficient experience data for end-to-end learning is often infeasible in the real world, as deploying robots in many environments can be prohibitively expensive. On the other hand, large-scale scene understanding datasets contain diverse and rich semantic and geometric information. But how to leverage such information for manipulation remains an open problem. In this paper, we propose a learning-to-plan method that can generalize to new object instances by leveraging object-level representations extracted from a synthetic scene understanding dataset. We evaluate our method with a suite of challenging multi-step manipulation tasks inspired by household activities and show that our model achieves measurably better success rate than state-of-the-art end-to-end approaches. Additional information can be found at https://sites.google.com/view/gentp