# Task and Motion Planning for Execution in the Real

arXiv: https://arxiv.org/abs/2406.03641 · HTML: https://mlsanity.com/paper/2406.03641/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 67 references that are in the map.
- First posted on arXiv 5 Jun 2024.

## Builds on (most cited first, 28 of 28)

- [End-to-End Training of Deep Visuomotor Policies](https://mlsanity.com/paper/1504.00702/) (2015)
- [Sim-to-Real Transfer in Deep Reinforcement Learning for Robotics: a Survey](https://mlsanity.com/paper/2009.13303/) (2020)
- [Integrated Task and Motion Planning](https://mlsanity.com/paper/2010.01083/) (2020)
- [Deep Object Pose Estimation for Semantic Robotic Grasping of Household Objects](https://mlsanity.com/paper/1809.10790/) (2018)
- [How to Train Your Robot with Deep Reinforcement Learning; Lessons We've Learned](https://mlsanity.com/paper/2102.02915/) (2021)
- [PDDLStream: Integrating Symbolic Planners and Blackbox Samplers via Optimistic Adaptive Planning](https://mlsanity.com/paper/1802.08705/) (2018)
- [Grasp Pose Detection in Point Clouds](https://mlsanity.com/paper/1706.09911/) (2017)
- [DESPOT: Online POMDP Planning with Regularization](https://mlsanity.com/paper/1609.03250/) (2016)
- [6-DoF Pose Estimation of Household Objects for Robotic Manipulation: An Accessible Dataset and Benchmark](https://mlsanity.com/paper/2203.05701/) (2022)
- [Neural Task Programming: Learning to Generalize Across Hierarchical Tasks](https://mlsanity.com/paper/1710.01813/) (2017)
- [Online Replanning in Belief Space for Partially Observable Task and Motion Problems](https://mlsanity.com/paper/1911.04577/) (2019)
- [Learning Neuro-Symbolic Skills for Bilevel Planning](https://mlsanity.com/paper/2206.10680/) (2022)
- [SORNet: Spatial Object-Centric Representations for Sequential Manipulation](https://mlsanity.com/paper/2109.03891/) (2021)
- [Long-Horizon Manipulation of Unknown Objects via Task and Motion Planning with Estimated Affordances](https://mlsanity.com/paper/2108.04145/) (2021)
- [Object-Centric Task and Motion Planning in Dynamic Environments](https://mlsanity.com/paper/1911.04679/) (2019)
- [Active Inference and Behavior Trees for Reactive Action Planning and Execution in Robotics](https://mlsanity.com/paper/2011.09756/) (2020)
- [Towards Blended Reactive Planning and Acting using Behavior Trees](https://mlsanity.com/paper/1611.00230/) (2016)
- [Transferable Task Execution from Pixels through Deep Planning Domain Learning](https://mlsanity.com/paper/2003.03726/) (2020)
- [Representing Robot Task Plans as Robust Logical-Dynamical Systems](https://mlsanity.com/paper/1908.01896/) (2019)
- [Anytime Integrated Task and Motion Policies for Stochastic Environments](https://mlsanity.com/paper/1904.13006/) (2019)
- [Behavior Trees in Robot Control Systems](https://mlsanity.com/paper/2203.13083/) (2022)
- [Hierarchical Policies for Cluttered-Scene Grasping with Latent Plans](https://mlsanity.com/paper/2107.01518/) (2021)
- [Active Learning of Abstract Plan Feasibility](https://mlsanity.com/paper/2107.00683/) (2021)
- [Reactive Task and Motion Planning under Temporal Logic Specifications](https://mlsanity.com/paper/2103.14464/) (2021)
- [Sequence-of-Constraints MPC: Reactive Timing-Optimal Control of Sequential Manipulation](https://mlsanity.com/paper/2203.05390/) (2022)
- [Robowflex: Robot Motion Planning with MoveIt Made Easy](https://mlsanity.com/paper/2103.12826/) (2021)
- [Probabilistic Inference in Planning for Partially Observable Long Horizon Problems](https://mlsanity.com/paper/2110.09153/) (2021)
- [Task and Motion Informed Trees (TMIT*): Almost-Surely Asymptotically Optimal Integrated Task and Motion Planning](https://mlsanity.com/paper/2210.09367/) (2022)

## Built on by (most cited first, 6 of 6)

- [ESearch-R1: Learning Cost-Aware MLLM Agents for Interactive Embodied Search via Reinforcement Learning](https://mlsanity.com/paper/2512.18571/) (2025)
- [Large-Language-Model-Guided State Estimation for Partially Observable Task and Motion Planning](https://mlsanity.com/paper/2603.03704/) (2026)
- [Task and Skill Planning: Hierarchical Robot Planning with Black-Box Skills](https://mlsanity.com/paper/2504.17901/) (2025)
- [Virtues of Ordered Chaos: Planning with Topple Actions in Tabletop Stack Rearrangement](https://mlsanity.com/paper/2605.17815/) (2026)
- [RE4: Transformation-aware Imitation of Object Interactions Using Manipulation Modes](https://mlsanity.com/paper/2606.24403/) (2026)
- [Neurosymbolic Reasoning with Incremental Knowledge for Sample Efficient Hierarchical Reinforcement Learning](https://mlsanity.com/paper/2608.02993/) (2026)

## Abstract (arXiv)

Task and motion planning represents a powerful set of hybrid planning methods that combine reasoning over discrete task domains and continuous motion generation. Traditional reasoning necessitates task domain models and enough information to ground actions to motion planning queries. Gaps in this knowledge often arise from sources like occlusion or imprecise modeling. This work generates task and motion plans that include actions cannot be fully grounded at planning time. During execution, such an action is handled by a provided human-designed or learned closed-loop behavior. Execution combines offline planned motions and online behaviors till reaching the task goal. Failures of behaviors are fed back as constraints to find new plans. Forty real-robot trials and motivating demonstrations are performed to evaluate the proposed framework and compare against state-of-the-art. Results show faster execution time, less number of actions, and more success in problems where diverse gaps arise. The experiment data is shared for researchers to simulate these settings. The work shows promise in expanding the applicable class of realistic partially grounded problems that robots can address.