# Self-Recovery Prompting: Promptable General Purpose Service Robot System with Foundation Models and Self-Recovery

arXiv: https://arxiv.org/abs/2309.14425 · HTML: https://mlsanity.com/paper/2309.14425/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 42 references that are in the map.
- First posted on arXiv 25 Sep 2023.

## Builds on (most cited first, 28 of 28)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://mlsanity.com/paper/2201.12086/) (2022)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [Do As I Can, Not As I Say: Grounding Language in Robotic Affordances](https://mlsanity.com/paper/2204.01691/) (2022)
- [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://mlsanity.com/paper/2107.13586/) (2021)
- [Code as Policies: Language Model Programs for Embodied Control](https://mlsanity.com/paper/2209.07753/) (2022)
- [Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers](https://mlsanity.com/paper/2301.02111/) (2023)
- [Detecting Twenty-thousand Classes using Image-level Supervision](https://mlsanity.com/paper/2201.02605/) (2022)
- [AudioCLIP: Extending CLIP to Image, Text and Audio](https://mlsanity.com/paper/2106.13043/) (2021)
- [Robots That Ask For Help: Uncertainty Alignment for Large Language Model Planners](https://mlsanity.com/paper/2307.01928/) (2023)
- [ChatGPT for Robotics: Design Principles and Model Abilities](https://mlsanity.com/paper/2306.17582/) (2023)
- [REFLECT: Summarizing Robot Experiences for Failure Explanation and Correction](https://mlsanity.com/paper/2306.15724/) (2023)
- [CLIP-Fields: Weakly Supervised Semantic Fields for Robotic Memory](https://mlsanity.com/paper/2210.05663/) (2022)
- [DoReMi: Grounding Language Model by Detecting and Recovering from Plan-Execution Misalignment](https://mlsanity.com/paper/2307.00329/) (2023)
- [Audio Visual Language Maps for Robot Navigation](https://mlsanity.com/paper/2303.07522/) (2023)
- [DRAGON: A Dialogue-Based Robot for Assistive Navigation with Visual Language Grounding](https://mlsanity.com/paper/2307.06924/) (2023)
- [World Robot Challenge 2020 -- Partner Robot: A Data-Driven Approach for Room Tidying with Mobile Manipulator](https://mlsanity.com/paper/2207.10106/) (2022)
- [Foundation Model based Open Vocabulary Task Planning and Executive System for General Purpose Service Robots](https://mlsanity.com/paper/2308.03357/) (2023)
- [Desiderata for Planning Systems in General-Purpose Service Robots](https://mlsanity.com/paper/1907.02300/) (2019)

## Built on by (most cited first, 6 of 6)

- [Real-World Robot Applications of Foundation Models: A Review](https://mlsanity.com/paper/2402.05741/) (2024)
- [Safety Control of Service Robots with LLMs and Embodied Knowledge Graphs](https://mlsanity.com/paper/2405.17846/) (2024)
- [Task Success Prediction for Open-Vocabulary Manipulation Based on Multi-Level Aligned Representations](https://mlsanity.com/paper/2410.00436/) (2024)
- [A Comprehensive Survey on Physical Risk Control in the Era of Foundation Model-enabled Robotics](https://mlsanity.com/paper/2505.12583/) (2025)
- [SPINE: Bridging the Cyber-Physical Gap with Agentic AI](https://mlsanity.com/paper/2607.13049/) (2026)
- [Design and Evaluation of LLM Chaining-Based Task Planning for General Purpose Service Robots](https://mlsanity.com/paper/2609.29043/) (2026)

## Abstract (arXiv)

A general-purpose service robot (GPSR), which can execute diverse tasks in various environments, requires a system with high generalizability and adaptability to tasks and environments. In this paper, we first developed a top-level GPSR system for worldwide competition (RoboCup@Home 2023) based on multiple foundation models. This system is both generalizable to variations and adaptive by prompting each model. Then, by analyzing the performance of the developed system, we found three types of failure in more realistic GPSR application settings: insufficient information, incorrect plan generation, and plan execution failure. We then propose the self-recovery prompting pipeline, which explores the necessary information and modifies its prompts to recover from failure. We experimentally confirm that the system with the self-recovery mechanism can accomplish tasks by resolving various failure cases. Supplementary videos are available at https://sites.google.com/view/srgpsr .