# Towards Safer Mobile Agents: Scalable Generation and Evaluation of Diverse Scenarios for VLMs

arXiv: https://arxiv.org/abs/2601.08470 · HTML: https://mlsanity.com/paper/2601.08470/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Driving VLMs and open-vocabulary detection, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 32 of its 35 references that are in the map.
- First posted on arXiv 13 Jan 2026.

## Placed in

- Multimodal LLMs / vision-language models / [Driving VLMs and open-vocabulary detection](https://mlsanity.com/mllm/driving-vlms-and-open-vocabulary-detection/)
- LLM agents and tool use / [Multimodal and embodied agents](https://mlsanity.com/agents/multimodal-and-embodied-agents/)

## Builds on (most cited first, 30 of 32)

- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [Qwen2.5-VL Technical Report](https://mlsanity.com/paper/2502.13923/) (2025)
- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [Scalability in Perception for Autonomous Driving: Waymo Open Dataset](https://mlsanity.com/paper/1912.04838/) (2019)
- [CARLA: An Open Urban Driving Simulator](https://mlsanity.com/paper/1711.03938/) (2017)
- [InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency](https://mlsanity.com/paper/2508.18265/) (2025)
- [Qwen-Image Technical Report](https://mlsanity.com/paper/2508.02324/) (2025)
- [FLUX.1 Kontext: Flow Matching for In-Context Image Generation and Editing in Latent Space](https://mlsanity.com/paper/2506.15742/) (2025)
- [KITTI-360: A Novel Dataset and Benchmarks for Urban Scene Understanding in 2D and 3D](https://mlsanity.com/paper/2109.13410/) (2021)
- [DriveLM: Driving with Graph Visual Question Answering](https://mlsanity.com/paper/2312.14150/) (2023)
- [Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAs](https://mlsanity.com/paper/2503.01743/) (2025)
- [DriveVLM: The Convergence of Autonomous Driving and Large Vision-Language Models](https://mlsanity.com/paper/2402.12289/) (2024)
- [NuScenes-QA: A Multi-modal Visual Question Answering Benchmark for Autonomous Driving Scenario](https://mlsanity.com/paper/2305.14836/) (2023)
- [Textual Explanations for Self-Driving Vehicles](https://mlsanity.com/paper/1807.11546/) (2018)
- [PaliGemma 2: A Family of Versatile VLMs for Transfer](https://mlsanity.com/paper/2412.03555/) (2024)
- [LingoQA: Visual Question Answering for Autonomous Driving](https://mlsanity.com/paper/2312.14115/) (2023)
- [Talk2Car: Taking Control of Your Self-Driving Car](https://mlsanity.com/paper/1909.10838/) (2019)
- [Socially Compliant Navigation Dataset (SCAND): A Large-Scale Dataset of Demonstrations for Social Navigation](https://mlsanity.com/paper/2203.15041/) (2022)
- [DRAMA: Joint Risk Localization and Captioning in Driving](https://mlsanity.com/paper/2209.10767/) (2022)
- [Are VLMs Ready for Autonomous Driving? An Empirical Study from the Reliability, Data, and Metric Perspectives](https://mlsanity.com/paper/2501.04003/) (2025)
- [Language Prompt for Autonomous Driving](https://mlsanity.com/paper/2309.04379/) (2023)
- [Rank2Tell: A Multimodal Driving Dataset for Joint Importance Ranking and Reasoning](https://mlsanity.com/paper/2309.06597/) (2023)
- [SURDS: Benchmarking Spatial Understanding and Reasoning in Driving Scenarios with Vision Language Models](https://mlsanity.com/paper/2411.13112/) (2024)
- [Embodied Scene Understanding for Vision Language Models via MetaVQA](https://mlsanity.com/paper/2501.09167/) (2025)
- [Fine-Grained Evaluation of Large Vision-Language Models in Autonomous Driving](https://mlsanity.com/paper/2503.21505/) (2025)
- [Social-LLaVA: Enhancing Robot Navigation through Human-Language Reasoning in Social Spaces](https://mlsanity.com/paper/2501.09024/) (2024)
- [SafeDriveRAG: Towards Safe Autonomous Driving with Knowledge Graph-based Retrieval-Augmented Generation](https://mlsanity.com/paper/2507.21585/) (2025)
- [ScVLM: Enhancing Vision-Language Model for Safety-Critical Event Understanding](https://mlsanity.com/paper/2410.00982/) (2024)
- [DriveQA: Passing the Driving Knowledge Test](https://mlsanity.com/paper/2508.21824/) (2025)
- [SocialNav-SUB: Benchmarking VLMs for Scene Understanding in Social Robot Navigation](https://mlsanity.com/paper/2509.08757/) (2025)

## Built on by (most cited first, 1 of 1)

- [A Local Agentic AI Framework to Generate Edge Case Scenarios for Safety Assurance](https://mlsanity.com/paper/2610.13479/) (2026)

## Abstract (arXiv)

Vision Language Models (VLMs) are increasingly deployed in autonomous vehicles and mobile systems, making it crucial to evaluate their ability to support safer decision-making in complex environments. However, existing benchmarks inadequately cover diverse hazardous situations, especially anomalous scenarios with spatio-temporal dynamics. While image editing models are a promising means to synthesize such hazards, it remains challenging to generate well-formulated scenarios that include moving, intrusive, and distant objects frequently observed in the real world. To address this gap, we introduce HazardForge, a scalable pipeline that leverages image editing models to generate these scenarios with layout decision algorithms, and validation modules. Using HazardForge, we construct MovSafeBench, a multiple-choice question (MCQ) benchmark comprising 7,254 images and corresponding QA pairs across 13 object categories, covering both normal and anomalous objects. Experiments using MovSafeBench show that VLM performance degrades notably under conditions including anomalous objects, with the largest drop in scenarios requiring nuanced motion understanding.