# Object-Centric Task and Motion Planning in Dynamic Environments

arXiv: https://arxiv.org/abs/1911.04679 · HTML: https://mlsanity.com/paper/1911.04679/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Nov 2019.

## Built on by (most cited first, 30 of 35)

- [ReKep: Spatio-Temporal Reasoning of Relational Keypoint Constraints for Robotic Manipulation](https://mlsanity.com/paper/2409.01652/) (2024)
- [RoboGen: Towards Unleashing Infinite Data for Automated Robot Learning via Generative Simulation](https://mlsanity.com/paper/2311.01455/) (2023)
- [VIOLA: Imitation Learning for Vision-Based Manipulation with Object Proposal Priors](https://mlsanity.com/paper/2210.11339/) (2022)
- [Gen2Sim: Scaling up Robot Learning in Simulation with Generative Models](https://mlsanity.com/paper/2310.18308/) (2023)
- [Learning Generalizable Manipulation Policies with Object-Centric 3D Representations](https://mlsanity.com/paper/2310.14386/) (2023)
- [Robot See Robot Do: Imitating Articulated Object Manipulation with Monocular 4D Reconstruction](https://mlsanity.com/paper/2409.18121/) (2024)
- [Vision-based Manipulation from Single Human Video with Open-World Object Graphs](https://mlsanity.com/paper/2405.20321/) (2024)
- [ControlVLA: Few-shot Object-centric Adaptation for Pre-trained Vision-Language-Action Models](https://mlsanity.com/paper/2506.16211/) (2025)
- [Energy-based Models are Zero-Shot Planners for Compositional Scene Rearrangement](https://mlsanity.com/paper/2304.14391/) (2023)
- [A Survey of Optimization-based Task and Motion Planning: From Classical To Learning Approaches](https://mlsanity.com/paper/2404.02817/) (2024)
- [Rethinking Progression of Memory State in Robotic Manipulation: An Object-Centric Perspective](https://mlsanity.com/paper/2511.11478/) (2025)
- [Generalizable Long-Horizon Manipulations with Large Language Models](https://mlsanity.com/paper/2310.02264/) (2023)
- [Object-Centric Instruction Augmentation for Robotic Manipulation](https://mlsanity.com/paper/2401.02814/) (2024)
- [Composing Pre-Trained Object-Centric Representations for Robotics From "What" and "Where" Foundation Models](https://mlsanity.com/paper/2404.13474/) (2024)
- [Language-Guided Object-Centric Diffusion Policy for Generalizable and Collision-Aware Robotic Manipulation](https://mlsanity.com/paper/2407.00451/) (2024)
- [Object-Centric Learning with Slot Mixture Module](https://mlsanity.com/paper/2311.04640/) (2023)
- [Are We Done with Object-Centric Learning?](https://mlsanity.com/paper/2504.07092/) (2025)
- [Learning to Search in Task and Motion Planning with Streams](https://mlsanity.com/paper/2111.13144/) (2021)
- [Learning from 10 Demos: Generalisable and Sample-Efficient Policy Learning with Oriented Affordance Frames](https://mlsanity.com/paper/2410.12124/) (2024)
- [GraspARL: Dynamic Grasping via Adversarial Reinforcement Learning](https://mlsanity.com/paper/2203.02119/) (2022)
- [Task and Motion Planning for Execution in the Real](https://mlsanity.com/paper/2406.03641/) (2024)
- [Closed Loop Interactive Embodied Reasoning for Robot Manipulation](https://mlsanity.com/paper/2404.15194/) (2024)
- [Generalizing Object-Centric Task-Axes Controllers using Keypoints](https://mlsanity.com/paper/2103.10524/) (2021)
- [Active Task Randomization: Learning Robust Skills via Unsupervised Generation of Diverse and Feasible Tasks](https://mlsanity.com/paper/2211.06134/) (2022)
- [Pick-and-place Manipulation Across Grippers Without Retraining: A Learning-optimization Diffusion Policy Approach](https://mlsanity.com/paper/2502.15613/) (2025)
- [Executable Analytic Concepts as the Missing Link Between VLM Insight and Precise Manipulation](https://mlsanity.com/paper/2510.07975/) (2025)
- [Symbolic Disentangled Representations for Images](https://mlsanity.com/paper/2412.19847/) (2024)
- [Inference-stage Adaptation-projection Strategy Adapts Diffusion Policy to Cross-manipulators Scenarios](https://mlsanity.com/paper/2509.11621/) (2025)
- [Assembly Planning from Observations under Physical Constraints](https://mlsanity.com/paper/2204.09616/) (2022)
- [Logic Learning from Demonstrations for Multi-step Manipulation Tasks in Dynamic Environments](https://mlsanity.com/paper/2404.16138/) (2024)

## Abstract (arXiv)

We address the problem of applying Task and Motion Planning (TAMP) in real world environments. TAMP combines symbolic and geometric reasoning to produce sequential manipulation plans, typically specified as joint-space trajectories, which are valid only as long as the environment is static and perception and control are highly accurate. In case of any changes in the environment, slow re-planning is required. We propose a TAMP algorithm that optimizes over Cartesian frames defined relative to target objects. The resulting plan then remains valid even if the objects are moving and can be executed by reactive controllers that adapt to these changes in real time. We apply our TAMP framework to a torque-controlled robot in a pick and place setting and demonstrate its ability to adapt to changing environments, inaccurate perception, and imprecise control, both in simulation and the real world.