Addressing Failures in Robotics using Vision-Based Language Models (VLMs) and Behavior Trees (BT)
On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models
Abstract (arXiv)
In this paper, we propose an approach that combines Vision Language Models (VLMs) and Behavior Trees (BTs) to address failures in robotics. Current robotic systems can handle known failures with pre-existing recovery strategies, but they are often ill-equipped to manage unknown failures or anomalies. We introduce VLMs as a monitoring tool to detect and identify failures during task execution. Additionally, VLMs generate missing conditions or skill templates that are then incorporated into the BT, ensuring the system can autonomously address similar failures in future tasks. We validate our approach through simulations in several failure scenarios.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. First posted on arXiv 3 Nov 2024.
Builds on
No reference list yet.
Built on by
Citing papers per year: 2025: 2.
2 papers.
Similar, not cited
Close in wording, no citation either way
- 2025I-FailSense: Towards General Robotic Failure Detection with Vision-Language Models
- 2024AHA: A Vision-Language-Model for Detecting and Reasoning Over Failures in Robotic Manipulation
- 2024Automating Robot Failure Recovery Using Vision-Language Models With Optimized Prompts
- 2025Guardian: Detecting Robotic Planning and Execution Errors with Vision-Language Models
- 2026FailBench: How Reliable are VLMs at Judging Robot Task Success?
- 2025FailSafe: Reasoning and Recovery from Failures in Vision-Language-Action Models
- 2025SAFE: Multitask Failure Detection for Vision-Language-Action Models
- 2026VLA-FAIL: Efficient Task Failure Detection for Finetuned Vision-Language-Action Models