# Embodied Hazard Mitigation using Vision-Language Models for Autonomous Mobile Robots

arXiv: https://arxiv.org/abs/2509.06768 · HTML: https://mlsanity.com/paper/2509.06768/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Driving VLMs and open-vocabulary detection, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 10 of its 22 references that are in the map.
- First posted on arXiv 8 Sep 2025.

## Placed in

- Multimodal LLMs / vision-language models / [Driving VLMs and open-vocabulary detection](https://mlsanity.com/mllm/driving-vlms-and-open-vocabulary-detection/)

## Builds on (most cited first, 10 of 10)

- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://mlsanity.com/paper/2201.12086/) (2022)
- [VLFM: Vision-Language Frontier Maps for Zero-Shot Semantic Navigation](https://mlsanity.com/paper/2312.03275/) (2023)
- [Explainable AI for Robot Failures: Generating Explanations that Improve User Assistance in Fault Recovery](https://mlsanity.com/paper/2101.01625/) (2021)
- [Explainable Reinforcement Learning via Model Transforms](https://mlsanity.com/paper/2209.12006/) (2022)
- [Evaluating Human-like Explanations for Robot Actions in Reinforcement Learning Scenarios](https://mlsanity.com/paper/2207.03214/) (2022)
- [LLM-Enhanced Path Planning: Safe and Efficient Autonomous Navigation with Instructional Inputs](https://mlsanity.com/paper/2412.02655/) (2024)
- [A Surrogate Model Framework for Explainable Autonomous Behaviour](https://mlsanity.com/paper/2305.19724/) (2023)
- [Enhancing Robot Explanation Capabilities through Vision-Language Models: a Preliminary Study by Interpreting Visual Inputs for Improved Human-Robot Interaction](https://mlsanity.com/paper/2404.09705/) (2024)
- [TalkWithMachines: Enhancing Human-Robot Interaction for Interpretable Industrial Robotics Through Large/Vision Language Models](https://mlsanity.com/paper/2412.15462/) (2024)

## Abstract (arXiv)

Autonomous robots operating in dynamic environments should identify and report anomalies. Embodying proactive mitigation improves safety and operational continuity. This paper presents a multimodal anomaly detection and mitigation system that integrates vision-language models and large language models to identify and report hazardous situations and conflicts in real-time. The proposed system enables robots to perceive, interpret, report, and if possible respond to urban and environmental anomalies through proactive detection mechanisms and automated mitigation actions. A key contribution in this paper is the integration of Hazardous and Conflict states into the robot's decision-making framework, where each anomaly type can trigger specific mitigation strategies. User studies (n = 30) demonstrated the effectiveness of the system in anomaly detection with 91.2% prediction accuracy and relatively low latency response times using edge-ai architecture.