mlsanityData to 7 Oct 2026

REFLECT: Summarizing Robot Experiences for Failure Explanation and Correction

arXiv 2306.15724 ↗First posted 27 Jun 2023cs.RO cs.AI cs.CLcited by 171

Not on the map. Why?

Cited by 29 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

The ability to detect and analyze failed executions automatically is crucial for an explainable and robust robotic system. Recently, Large Language Models (LLMs) have demonstrated strong reasoning abilities on textual inputs. To leverage the power of LLMs for robot failure explanation, we introduce REFLECT, a framework which queries LLM for failure reasoning based on a hierarchical summary of robot past experiences generated from multisensory observations. The failure explanation can further guide a language-based planner to correct the failure and complete the task. To systematically evaluate the framework, we create the RoboFail dataset with a variety of tasks and failure scenarios. We demonstrate that the LLM-based framework is able to generate informative failure explanations that assist successful correction planning.

Key facts. Cited by 171 papers in the map (data through 7 Oct 2026). Builds on 36 of its 49 references that are in the map. First posted on arXiv 27 Jun 2023.

Builds on

36 of 49 references.

References · 36

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  5. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  6. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  7. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  8. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  9. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  10. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,172
  11. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,057
  12. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,029
24 more
  1. 2017AI2-THOR: An Interactive 3D Environment for Visual AIcited by 850
  2. 2021MDETR -- Modulated Detection for End-to-End Multi-Modal Understandingcited by 811
  3. 2022ProgPrompt: Generating Situated Robot Task Plans using Large Language Modelscited by 598
  4. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
  5. 2022LLM-Planner: Few-Shot Grounded Planning for Embodied Agents with Large Language Modelscited by 459
  6. 2021AudioCLIP: Extending CLIP to Image, Text and Audiocited by 335
  7. 2022Visual Spatial Reasoningcited by 288
  8. 2023Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioningcited by 263
  9. 2021Wav2CLIP: Learning Robust Audio Representations From CLIPcited by 210
  10. 2023Generalized Planning in PDDL Domains with Pretrained Large Language Modelscited by 139
  11. 2021End-to-End Dense Video Captioning with Parallel Decodingcited by 133
  12. 2022Language Models with Image Descriptors are Strong Few-Shot Video-Language Learnerscited by 117
  13. 2023Errors are Useful Prompts: Instruction Guided Task Programming with Verifier-Assisted Iterative Promptingcited by 45
  14. 2021Explainable AI for Robot Failures: Generating Explanations that Improve User Assistance in Fault Recoverycited by 32
  15. 2022CAPE: Corrective Actions from Precondition Errors using Large Language Modelscited by 30
  16. 2022End-to-end Dense Video Captioning as Sequence Generationcited by 27
  17. 2023Summarize the Past to Predict the Future: Natural Language Descriptions of Context Boost Multimodal Object Interaction Anticipationcited by 21
  18. 2023Modeling Dynamic Environments with Scene Graph Memorycited by 18
  19. 2022Unifying Event Detection and Captioning as Sequence Generation via Pre-Trainingcited by 17
  20. 2020FINO-Net: A Deep Multimodal Sensor Fusion Framework for Manipulation Failure Detectioncited by 13
  21. 2022Why did I fail? A Causal-based Method to Find Explanations for Robot Failurescited by 13
  22. 2021Semantic-Based Explainable AI: Leveraging Semantic Scene Graphs and Pairwise Ranking to Explain Robot Failurescited by 12
  23. 2023User Study Exploring the Role of Explanation of Failures by Robots in Human Robot Collaboration Taskscited by 5
  24. 2022Summarizing a virtual robot's past actions in natural languagecited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 15; 2024: 51; 2025: 47; 2026: 58.

171 papers.

Sort:

By area:

  1. 2023ExpeL: LLM Agents Are Experiential Learnerscited by 814
  2. 2024PIVOT: Iterative Visual Prompting Elicits Actionable Knowledge for VLMscited by 187
  3. 2024Scene-LLM: Extending Language Model for 3D Visual Understanding and Reasoningcited by 170
  4. 2024RoboMIND: Benchmark on Multi-embodiment Intelligence Normative Data for Robot Manipulationcited by 166
  5. 2025Cosmos-Reason1: From Physical Common Sense To Embodied Reasoningcited by 159
  6. 2025ThinkAct: Vision-Language-Action Reasoning via Reinforced Visual Latent Planningcited by 152
  7. 2025RoboBrain: A Unified Brain Model for Robotic Manipulation from Abstract to Concretecited by 134
  8. 2023Large Language Models for Robotics: A Surveycited by 132
  9. 2023GenSim: Generating Robotic Simulation Tasks via Large Language Modelscited by 120
  10. 2024Meta-Prompting: Enhancing Language Models with Task-Agnostic Scaffoldingcited by 115
  11. 2023Avalon's Game of Thoughts: Battle Against Deception through Recursive Contemplationcited by 102
  12. 2023Toward General-Purpose Robots via Foundation Models: A Survey and Meta-Analysiscited by 99
38 more
  1. 2024Large Language Models for Robotics: Opportunities, Challenges, and Perspectivescited by 88
  2. 2024Unpacking Failure Modes of Generative Policies: Runtime Monitoring of Consistency and Progresscited by 63
  3. 2024Real-World Robot Applications of Foundation Models: A Reviewcited by 60
  4. 2024Code-as-Monitor: Constraint-aware Visual Programming for Reactive and Proactive Robotic Failure Detectioncited by 59
  5. 2024VLM Agents Generate Their Own Memories: Distilling Experience into Embodied Programs of Thoughtcited by 55
  6. 2025From Foresight to Forethought: VLM-In-the-Loop Policy Steering via Latent Alignmentcited by 51
  7. 2024TRANSIC: Sim-to-Real Policy Transfer by Learning from Online Correctioncited by 50
  8. 2024RACER: Rich Language-Guided Failure Recovery Policies for Imitation Learningcited by 46
  9. 2023Distilling and Retrieving Generalizable Knowledge for Robot Manipulation via Language Correctionscited by 44
  10. 2024Robot Utility Models: General Policies for Zero-Shot Deployment in New Environmentscited by 43
  11. 2024On the Prospects of Incorporating Large Language Models (LLMs) in Automated Planning and Scheduling (APS)cited by 40
  12. 2024Closed-Loop Open-Vocabulary Mobile Manipulation with GPT-4Vcited by 40
  13. 2023Open-Ended Instructable Embodied Agents with Memory-Augmented Large Language Modelscited by 37
  14. 2025RoboFAC: A Comprehensive Framework for Robotic Failure Analysis and Correctioncited by 37
  15. 2023Interactive Task Planning with Language Modelscited by 36
  16. 2025Failure Prediction at Runtime for Generative Robot Policiescited by 36
  17. 2024V-IRL: Grounding Virtual Intelligence in Real Lifecited by 35
  18. 2024Multi-Task Interactive Robot Fleet Learning with Visual World Modelscited by 33
  19. 2025PAI-Bench: A Comprehensive Benchmark For Physical AIcited by 30
  20. 2024ManiWAV: Learning Robot Manipulation from In-the-Wild Audio-Visual Datacited by 29
  21. 2023Interactive Planning Using Large Language Models for Partially Observable Robotics Taskscited by 27
  22. 2024Introspective Planning: Aligning Robots' Uncertainty with Inherent Task Ambiguitycited by 26
  23. 2025CorrectNav: Self-Correction Flywheel Empowers Vision-Language-Action Navigation Modelcited by 26
  24. 2025Robobench: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models as Embodied Braincited by 24
  25. 2024Task Success is not Enough: Investigating the Use of Video-Language Models as Behavior Critics for Catching Undesirable Agent Behaviorscited by 23
  26. 2025Embodied AI: From LLMs to World Modelscited by 23
  27. 2023LLM for SoC Security: A Paradigm Shiftcited by 22
  28. 2024LLM-Driven Robots Risk Enacting Discrimination, Violence, and Unlawful Actionscited by 21
  29. 2025Counterfactual VLA: Self-Reflective Vision-Language-Action Model with Adaptive Reasoningcited by 21
  30. 2024A Self-Correcting Vision-Language-Action Model for Fast and Slow System Manipulationcited by 20
  31. 2024AIC MLLM: Autonomous Interactive Correction MLLM for Robust Robotic Manipulationcited by 20
  32. 2024A Survey of Optimization-based Task and Motion Planning: From Classical To Learning Approachescited by 19
  33. 2024Learning Reward for Robot Skills Using Large Language Models via Self-Alignmentcited by 19
  34. 2025Building a Mind Palace: Structuring Environment-Grounded Semantic Graphs for Effective Long Video Analysis with LLMscited by 19
  35. 2024MALMM: Multi-Agent Large Language Models for Zero-Shot Robotics Manipulationcited by 18
  36. 2026Show-Harness: Just a VLM Agent Can Play Robotscited by 17
  37. 2023Generalizable Long-Horizon Manipulations with Large Language Modelscited by 16
  38. 2024Automating Robot Failure Recovery Using Vision-Language Models With Optimized Promptscited by 16

50 most cited of 171 All 171 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024AHA: A Vision-Language-Model for Detecting and Reasoning Over Failures in Robotic Manipulation
  2. 2025I-FailSense: Towards General Robotic Failure Detection with Vision-Language Models
  3. 2025FailSafe: Reasoning and Recovery from Failures in Vision-Language-Action Models
  4. 2024Addressing Failures in Robotics using Vision-Based Language Models (VLMs) and Behavior Trees (BT)
  5. 2026RAFAIL: Relationship-Aware Failure Detection for Robotic Manipulation ↗
  6. 2026Foresight: Failure Detection for Long-Horizon Robotic Manipulation with Action-Conditioned World Model Latents
  7. 2026RoboProcessBench: Benchmarking Process-Aware Understanding in Vision-Language Robotic Manipulation
  8. 2025Can We Detect Failures Without Failure Data? Uncertainty-Aware Runtime Failure Detection for Imitation Learning Policies