# Trust the PRoC3S: Solving Long-Horizon Robotics Problems with LLMs and Constraint Satisfaction

arXiv: https://arxiv.org/abs/2406.05572 · HTML: https://mlsanity.com/paper/2406.05572/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 38 references that are in the map.
- First posted on arXiv 8 Jun 2024.

## Builds on (most cited first, 28 of 28)

- [Do As I Can, Not As I Say: Grounding Language in Robotic Affordances](https://mlsanity.com/paper/2204.01691/) (2022)
- [Code as Policies: Language Model Programs for Embodied Control](https://mlsanity.com/paper/2209.07753/) (2022)
- [Inner Monologue: Embodied Reasoning through Planning with Language Models](https://mlsanity.com/paper/2207.05608/) (2022)
- [Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents](https://mlsanity.com/paper/2201.07207/) (2022)
- [VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Models](https://mlsanity.com/paper/2307.05973/) (2023)
- [ProgPrompt: Generating Situated Robot Task Plans using Large Language Models](https://mlsanity.com/paper/2209.11302/) (2022)
- [Transporter Networks: Rearranging the Visual World for Robotic Manipulation](https://mlsanity.com/paper/2010.14406/) (2020)
- [Integrated Task and Motion Planning](https://mlsanity.com/paper/2010.01083/) (2020)
- [Text2Motion: From Natural Language Instructions to Feasible Plans](https://mlsanity.com/paper/2303.12153/) (2023)
- [LLMs Can't Plan, But Can Help Planning in LLM-Modulo Frameworks](https://mlsanity.com/paper/2402.01817/) (2024)
- [PDDLStream: Integrating Symbolic Planners and Blackbox Samplers via Optimistic Adaptive Planning](https://mlsanity.com/paper/1802.08705/) (2018)
- [Task and Motion Planning with Large Language Models for Object Rearrangement](https://mlsanity.com/paper/2303.06247/) (2023)
- [AutoTAMP: Autoregressive Task and Motion Planning with LLMs as Translators and Checkers](https://mlsanity.com/paper/2306.06531/) (2023)
- [SatLM: Satisfiability-Aided Language Models Using Declarative Prompting](https://mlsanity.com/paper/2305.09656/) (2023)
- [Benchmarking in Manipulation Research: The YCB Object and Model Set and Benchmarking Protocols](https://mlsanity.com/paper/1502.03143/) (2015)
- [CoPa: General Robotic Manipulation through Spatial Constraints of Parts with Foundation Models](https://mlsanity.com/paper/2403.08248/) (2024)
- [Grounded Decoding: Guiding Text Generation with Grounded Models for Embodied Agents](https://mlsanity.com/paper/2303.00855/) (2023)
- [LLM3:Large Language Model-based Task and Motion Planning with Motion Failure Reasoning](https://mlsanity.com/paper/2403.11552/) (2024)
- [DoReMi: Grounding Language Model by Detecting and Recovering from Plan-Execution Misalignment](https://mlsanity.com/paper/2307.00329/) (2023)
- [SayCanPay: Heuristic Planning with Large Language Models using Learnable Domain Knowledge](https://mlsanity.com/paper/2308.12682/) (2023)
- [Large Language Models Can Solve Real-World Planning Rigorously with Formal Verification Tools](https://mlsanity.com/paper/2404.11891/) (2024)
- [Learning Neuro-Symbolic Relational Transition Models for Bilevel Planning](https://mlsanity.com/paper/2105.14074/) (2021)
- [RePLan: Robotic Replanning with Perception and Language Models](https://mlsanity.com/paper/2401.04157/) (2024)
- [Long-Horizon Manipulation of Unknown Objects via Task and Motion Planning with Estimated Affordances](https://mlsanity.com/paper/2108.04145/) (2021)
- [Compositional Diffusion-Based Continuous Constraint Solvers](https://mlsanity.com/paper/2309.00966/) (2023)
- [Practice Makes Perfect: Planning to Learn Skill Parameter Policies](https://mlsanity.com/paper/2402.15025/) (2024)
- [STAP: Sequencing Task-Agnostic Policies](https://mlsanity.com/paper/2210.12250/) (2022)
- [On Classifying Continuous Constraint Satisfaction Problems](https://mlsanity.com/paper/2106.02397/) (2021)

## Built on by (most cited first, 19 of 19)

- [PlanGenLLMs: A Modern Survey of LLM Planning Capabilities](https://mlsanity.com/paper/2502.11221/) (2025)
- [QuestBench: Can LLMs ask the right question to acquire information in reasoning tasks?](https://mlsanity.com/paper/2503.22674/) (2025)
- [LLM-Guided Probabilistic Program Induction for POMDP Model Estimation](https://mlsanity.com/paper/2505.02216/) (2025)
- [From Pixels to Predicates: Learning Symbolic World Models via Pretrained Vision-Language Models](https://mlsanity.com/paper/2501.00296/) (2024)
- [MCP-Solver: Integrating Language Models with Constraint Programming Systems](https://mlsanity.com/paper/2501.00539/) (2024)
- [Code-as-Symbolic-Planner: Foundation Model-Based Robot Planning via Symbolic Code Generation](https://mlsanity.com/paper/2503.01700/) (2025)
- [LASP: Surveying the State-of-the-Art in Large Language Model-Assisted AI Planning](https://mlsanity.com/paper/2409.01806/) (2024)
- [Open-World Task and Motion Planning via Vision-Language Model Generated Constraints](https://mlsanity.com/paper/2411.08253/) (2024)
- [End-to-End Dexterous Arm-Hand VLA Policies via Shared Autonomy: VR Teleoperation Augmented by Autonomous Hand VLA Policy for Efficient Data Collection](https://mlsanity.com/paper/2511.00139/) (2025)
- [A Systematic Study of Large Language Models for Task and Motion Planning With PDDLStream](https://mlsanity.com/paper/2510.00182/) (2025)
- [SAIL: Test-Time Scaling for In-Context Imitation Learning with VLM](https://mlsanity.com/paper/2603.08269/) (2026)
- [Large-Language-Model-Guided State Estimation for Partially Observable Task and Motion Planning](https://mlsanity.com/paper/2603.03704/) (2026)
- [Coding Agents for Generalized Task and Motion Planning Problems](https://mlsanity.com/paper/2609.30233/) (2026)
- [Coloring Between the Lines: Personalization in the Null Space of Planning Constraints](https://mlsanity.com/paper/2505.15503/) (2025)
- ["Don't Do That!": Guiding Embodied Systems through Large Language Model-based Constraint Generation](https://mlsanity.com/paper/2506.04500/) (2025)
- [GAVEL: Graph World Models for Verified and Efficient Long-Horizon LLM Task Planning](https://mlsanity.com/paper/2609.19315/) (2026)
- [Generalizing Manipulation Skills with a Local Coding Agent](https://mlsanity.com/paper/2609.26499/) (2026)
- [Rational Inverse Reasoning: Few-Shot Imitation by Inferring Intent through Planning](https://mlsanity.com/paper/2508.08983/) (2025)
- [ManiLong-Shot: Interaction-Aware One-Shot Imitation Learning for Long-Horizon Manipulation](https://mlsanity.com/paper/2512.16302/) (2025)

## Abstract (arXiv)

Recent developments in pretrained large language models (LLMs) applied to robotics have demonstrated their capacity for sequencing a set of discrete skills to achieve open-ended goals in simple robotic tasks. In this paper, we examine the topic of LLM planning for a set of continuously parameterized skills whose execution must avoid violations of a set of kinematic, geometric, and physical constraints. We prompt the LLM to output code for a function with open parameters, which, together with environmental constraints, can be viewed as a Continuous Constraint Satisfaction Problem (CCSP). This CCSP can be solved through sampling or optimization to find a skill sequence and continuous parameter settings that achieve the goal while avoiding constraint violations. Additionally, we consider cases where the LLM proposes unsatisfiable CCSPs, such as those that are kinematically infeasible, dynamically unstable, or lead to collisions, and re-prompt the LLM to form a new CCSP accordingly. Experiments across three different simulated 3D domains demonstrate that our proposed strategy, PRoC3S, is capable of solving a wide range of complex manipulation tasks with realistic constraints on continuous parameters much more efficiently and effectively than existing baselines.