# A Survey of Optimization-based Task and Motion Planning: From Classical To Learning Approaches

arXiv: https://arxiv.org/abs/2404.02817 · HTML: https://mlsanity.com/paper/2404.02817/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 153 of its 282 references that are in the map.
- First posted on arXiv 3 Apr 2024.

## Builds on (most cited first, 30 of 153)

- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [Diffusion Policy: Visuomotor Policy Learning via Action Diffusion](https://mlsanity.com/paper/2303.04137/) (2023)
- [Do As I Can, Not As I Say: Grounding Language in Robotic Affordances](https://mlsanity.com/paper/2204.01691/) (2022)
- [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://mlsanity.com/paper/2107.13586/) (2021)
- [Inner Monologue: Embodied Reasoning through Planning with Language Models](https://mlsanity.com/paper/2207.05608/) (2022)
- [Planning with Diffusion for Flexible Behavior Synthesis](https://mlsanity.com/paper/2205.09991/) (2022)
- [Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents](https://mlsanity.com/paper/2201.07207/) (2022)
- [VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Models](https://mlsanity.com/paper/2307.05973/) (2023)
- [SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilities](https://mlsanity.com/paper/2401.12168/) (2024)
- [Offline Reinforcement Learning as One Big Sequence Modeling Problem](https://mlsanity.com/paper/2106.02039/) (2021)
- [Machine Learning for Combinatorial Optimization: a Methodological Tour d'Horizon](https://mlsanity.com/paper/1811.06128/) (2018)
- [ProgPrompt: Generating Situated Robot Task Plans using Large Language Models](https://mlsanity.com/paper/2209.11302/) (2022)
- [Is Conditional Generative Modeling all you need for Decision-Making?](https://mlsanity.com/paper/2211.15657/) (2022)
- [LLM+P: Empowering Large Language Models with Optimal Planning Proficiency](https://mlsanity.com/paper/2304.11477/) (2023)
- [Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimization](https://mlsanity.com/paper/1603.00448/) (2016)
- [Exact Combinatorial Optimization with Graph Convolutional Neural Networks](https://mlsanity.com/paper/1906.01629/) (2019)
- [Language to Rewards for Robotic Skill Synthesis](https://mlsanity.com/paper/2306.08647/) (2023)
- [Robots That Ask For Help: Uncertainty Alignment for Large Language Model Planners](https://mlsanity.com/paper/2307.01928/) (2023)
- [SayPlan: Grounding Large Language Models using 3D Scene Graphs for Scalable Robot Task Planning](https://mlsanity.com/paper/2307.06135/) (2023)
- [Integrated Task and Motion Planning](https://mlsanity.com/paper/2010.01083/) (2020)
- [Sustainable AI: Environmental Implications, Challenges and Opportunities](https://mlsanity.com/paper/2111.00364/) (2021)
- [MathPrompter: Mathematical Reasoning using Large Language Models](https://mlsanity.com/paper/2303.05398/) (2023)
- [Large Language Models as Commonsense Knowledge for Large-Scale Task Planning](https://mlsanity.com/paper/2305.14078/) (2023)
- [Text2Motion: From Natural Language Instructions to Feasible Plans](https://mlsanity.com/paper/2303.12153/) (2023)
- [Augmenting Language Models with Long-Term Memory](https://mlsanity.com/paper/2306.07174/) (2023)
- [Real-World Humanoid Locomotion with Reinforcement Learning](https://mlsanity.com/paper/2303.03381/) (2023)
- [REFLECT: Summarizing Robot Experiences for Failure Explanation and Correction](https://mlsanity.com/paper/2306.15724/) (2023)
- [Vision-Language Models are Zero-Shot Reward Models for Reinforcement Learning](https://mlsanity.com/paper/2310.12921/) (2023)
- [Open-vocabulary Queryable Scene Representations for Real World Planning](https://mlsanity.com/paper/2209.09874/) (2022)

## Built on by (most cited first, 20 of 20)

- [Aligning Cyber Space with Physical World: A Comprehensive Survey on Embodied AI](https://mlsanity.com/paper/2407.06886/) (2024)
- [LLMs as Planning Formalizers: A Survey for Leveraging Large Language Models to Construct Automated Planning Models](https://mlsanity.com/paper/2503.18971/) (2025)
- [Multi-agent Embodied AI: Advances and Future Directions](https://mlsanity.com/paper/2505.05108/) (2025)
- [Dream2Flow: Bridging Video Generation and Open-World Manipulation with 3D Object Flow](https://mlsanity.com/paper/2512.24766/) (2025)
- [ASHiTA: Automatic Scene-grounded HIerarchical Task Analysis](https://mlsanity.com/paper/2504.06553/) (2025)
- [CHD: Coupled Hierarchical Diffusion for Long-Horizon Tasks](https://mlsanity.com/paper/2505.07261/) (2025)
- [Using large language models for embodied planning introduces systematic safety risks](https://mlsanity.com/paper/2604.18463/) (2026)
- [VernaCopter: Disambiguated Natural-Language-Driven Robot via Formal Specifications](https://mlsanity.com/paper/2409.09536/) (2024)
- [Vision-Language-Policy Model for Dynamic Robot Task Planning](https://mlsanity.com/paper/2512.19178/) (2025)
- [MuTRAP: Multi-trigger Trojans Attacking Robot Task Planning Systems](https://mlsanity.com/paper/2504.17070/) (2025)
- [IMBench: A Benchmark for Intuitive Robotic Manipulation](https://mlsanity.com/paper/2607.15641/) (2026)
- [From Motion to Behavior: Hierarchical Modeling of Humanoid Generative Behavior Control](https://mlsanity.com/paper/2506.00043/) (2025)
- [LADY: Linear Attention for Autonomous Driving Efficiency without Transformers](https://mlsanity.com/paper/2512.15038/) (2025)
- [SIT-LMPC: Safe Information-Theoretic Learning Model Predictive Control for Iterative Tasks](https://mlsanity.com/paper/2602.16187/) (2026)
- [A Hierarchical Error-Corrective Graph Framework for Autonomous Agents with LLM-Based Action Generation](https://mlsanity.com/paper/2603.08388/) (2026)
- [Learning Bilevel Policies over Symbolic World Models for Long-Horizon Planning](https://mlsanity.com/paper/2605.15975/) (2026)
- [Bounding Boxes as Goals: Language-Conditioned Grasping via Neuro-Symbolic Planning](https://mlsanity.com/paper/2606.12910/) (2026)
- [MagicSim: A Unified Infrastructure for Executable Embodied Interaction](https://mlsanity.com/paper/2606.17511/) (2026)
- [Human2Any: Human-to-Robot Transfer via Constraint-Aware Compositional Planning](https://mlsanity.com/paper/2606.28813/) (2026)
- [Safe Task Planning with Long-Term Graph Memory for Embodied Agents](https://mlsanity.com/paper/2609.08444/) (2026)

## Abstract (arXiv)

Task and Motion Planning (TAMP) integrates high-level task planning and low-level motion planning to equip robots with the autonomy to effectively reason over long-horizon, dynamic tasks. Optimization-based TAMP focuses on hybrid optimization approaches that define goal conditions via objective functions and are capable of handling open-ended goals, robotic dynamics, and physical interaction between the robot and the environment. Therefore, optimization-based TAMP is particularly suited to solve highly complex, contact-rich locomotion and manipulation problems. This survey provides a comprehensive review on optimization-based TAMP, covering (i) planning domain representations, including action description languages and temporal logic, (ii) individual solution strategies for components of TAMP, including AI planning and trajectory optimization (TO), and (iii) the dynamic interplay between logic-based task planning and model-based TO. A particular focus of this survey is to highlight the algorithm structures to efficiently solve TAMP, especially hierarchical and distributed approaches. Additionally, the survey emphasizes the synergy between the classical methods and contemporary learning-based innovations such as large language models. Furthermore, the future research directions for TAMP is discussed in this survey, highlighting both algorithmic and application-specific challenges.