# Errors are Useful Prompts: Instruction Guided Task Programming with Verifier-Assisted Iterative Prompting

arXiv: https://arxiv.org/abs/2303.14100 · HTML: https://mlsanity.com/paper/2303.14100/

## Key facts

- Cited by 46 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Mar 2023.

## Built on by (most cited first, 30 of 46)

- [Voyager: An Open-Ended Embodied Agent with Large Language Models](https://mlsanity.com/paper/2305.16291/) (2023)
- [MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework](https://mlsanity.com/paper/2308.00352/) (2023)
- [SayPlan: Grounding Large Language Models using 3D Scene Graphs for Scalable Robot Task Planning](https://mlsanity.com/paper/2307.06135/) (2023)
- [OctoPack: Instruction Tuning Code Large Language Models](https://mlsanity.com/paper/2308.07124/) (2023)
- [REFLECT: Summarizing Robot Experiences for Failure Explanation and Correction](https://mlsanity.com/paper/2306.15724/) (2023)
- [GenSim: Generating Robotic Simulation Tasks via Large Language Models](https://mlsanity.com/paper/2310.01361/) (2023)
- [AutoTAMP: Autoregressive Task and Motion Planning with LLMs as Translators and Checkers](https://mlsanity.com/paper/2306.06531/) (2023)
- [Foundational Models Defining a New Era in Vision: A Survey and Outlook](https://mlsanity.com/paper/2307.13721/) (2023)
- [Code-as-Monitor: Constraint-aware Visual Programming for Reactive and Proactive Robotic Failure Detection](https://mlsanity.com/paper/2412.04455/) (2024)
- [ORGANA: A Robotic Assistant for Automated Chemistry Experimentation and Characterization](https://mlsanity.com/paper/2401.06949/) (2024)
- [Physics of Language Models: Part 2.2, How to Learn From Mistakes on Grade-School Math Problems](https://mlsanity.com/paper/2408.16293/) (2024)
- [LLM Collaboration With Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2508.04652/) (2025)
- [Interactive Task Planning with Language Models](https://mlsanity.com/paper/2310.10645/) (2023)
- [Steering Large Language Models between Code Execution and Textual Reasoning](https://mlsanity.com/paper/2410.03524/) (2024)
- [A Self-Correcting Vision-Language-Action Model for Fast and Slow System Manipulation](https://mlsanity.com/paper/2405.17418/) (2024)
- [AIC MLLM: Autonomous Interactive Correction MLLM for Robust Robotic Manipulation](https://mlsanity.com/paper/2406.11548/) (2024)
- [Incremental Learning of Humanoid Robot Behavior from Natural Interaction and Large Language Models](https://mlsanity.com/paper/2309.04316/) (2023)
- [LangProp: A code optimization framework using Large Language Models applied to driving](https://mlsanity.com/paper/2401.10314/) (2024)
- [LLM-State: Open World State Representation for Long-horizon Task Planning with Large Language Model](https://mlsanity.com/paper/2311.17406/) (2023)
- [Evaluating Uncertainty-based Failure Detection for Closed-Loop LLM Planners](https://mlsanity.com/paper/2406.00430/) (2024)
- [Sasha: Creative Goal-Oriented Reasoning in Smart Homes with Large Language Models](https://mlsanity.com/paper/2305.09802/) (2023)
- [Cohesive Conversations: Enhancing Authenticity in Multi-Agent Simulated Dialogues](https://mlsanity.com/paper/2407.09897/) (2024)
- [QCalEval: Benchmarking Vision-Language Models for Quantum Calibration Plot Understanding](https://mlsanity.com/paper/2604.25884/) (2026)
- [Code-as-Symbolic-Planner: Foundation Model-Based Robot Planning via Symbolic Code Generation](https://mlsanity.com/paper/2503.01700/) (2025)
- [Selective Perception: Optimizing State Descriptions with Reinforcement Learning for Language Model Actors](https://mlsanity.com/paper/2307.11922/) (2023)
- [ReFF: Reinforcing Format Faithfulness in Language Models across Varied Tasks](https://mlsanity.com/paper/2412.09173/) (2024)
- [Hierarchical Planning for Complex Tasks with Knowledge Graph-RAG and Symbolic Verification](https://mlsanity.com/paper/2504.04578/) (2025)
- [ConceptAgent: LLM-Driven Precondition Grounding and Tree Search for Robust Task Planning and Execution](https://mlsanity.com/paper/2410.06108/) (2024)
- [Learning Game-Playing Agents with Generative Code Optimization](https://mlsanity.com/paper/2508.19506/) (2025)
- [Learning Decentralized LLM Collaboration with Multi-Agent Actor Critic](https://mlsanity.com/paper/2601.21972/) (2026)

## Abstract (arXiv)

Generating low-level robot task plans from high-level natural language instructions remains a challenging problem. Although large language models have shown promising results in generating plans, the accuracy of the output remains unverified. Furthermore, the lack of domain-specific language data poses a limitation on the applicability of these models. In this paper, we propose CLAIRIFY, a novel approach that combines automatic iterative prompting with program verification to ensure programs written in data-scarce domain-specific language are syntactically valid and incorporate environment constraints. Our approach provides effective guidance to the language model on generating structured-like task plans by incorporating any errors as feedback, while the verifier ensures the syntactic accuracy of the generated plans. We demonstrate the effectiveness of CLAIRIFY in planning chemistry experiments by achieving state-of-the-art results. We also show that the generated plans can be executed on a real robot by integrating them with a task and motion planner.