# mmWalk: Towards Multi-modal Multi-view Walking Assistance

arXiv: https://arxiv.org/abs/2510.11520 · HTML: https://mlsanity.com/paper/2510.11520/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 57 references that are in the map.
- First posted on arXiv 13 Oct 2025.

## Builds on (most cited first, 23 of 23)

- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [LLaVA-OneVision: Easy Visual Task Transfer](https://mlsanity.com/paper/2408.03326/) (2024)
- [InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks](https://mlsanity.com/paper/2312.14238/) (2023)
- [CARLA: An Open Urban Driving Simulator](https://mlsanity.com/paper/1711.03938/) (2017)
- [Ego4D: Around the World in 3,000 Hours of Egocentric Video](https://mlsanity.com/paper/2110.07058/) (2021)
- [Chameleon: Mixed-Modal Early-Fusion Foundation Models](https://mlsanity.com/paper/2405.09818/) (2024)
- [VizWiz Grand Challenge: Answering Visual Questions from Blind People](https://mlsanity.com/paper/1802.08218/) (2018)
- [Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scaling](https://mlsanity.com/paper/2501.17811/) (2025)
- [LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methods](https://mlsanity.com/paper/2412.05579/) (2024)
- [Preference Leakage: A Contamination Problem in LLM-as-a-judge](https://mlsanity.com/paper/2502.01534/) (2025)
- [EgoTextVQA: Towards Egocentric Scene-Text Aware Video Question Answering](https://mlsanity.com/paper/2502.07411/) (2025)
- [SpatialLLM: A Compound 3D-Informed Design towards Spatially-Intelligent Large Multimodal Models](https://mlsanity.com/paper/2505.00788/) (2025)
- [Toward Human-Like Social Robot Navigation: A Large-Scale, Multi-Modal, Social Human Navigation Dataset](https://mlsanity.com/paper/2303.14880/) (2023)
- [NavAgent: Multi-scale Urban Street View Fusion For UAV Embodied Vision-and-Language Navigation](https://mlsanity.com/paper/2411.08579/) (2024)
- [SANPO: A Scene Understanding, Accessibility and Human Navigation Dataset](https://mlsanity.com/paper/2309.12172/) (2023)
- [GuideDog: A Real-World Egocentric Multimodal Dataset for Blind and Low-Vision Accessibility-Aware Guidance](https://mlsanity.com/paper/2503.12844/) (2025)
- [StreetNav: Leveraging Street Cameras to Support Precise Outdoor Navigation for Blind Pedestrians](https://mlsanity.com/paper/2310.00491/) (2023)
- [Space-Aware Instruction Tuning: Dataset and Benchmark for Guide Dog Robots Assisting the Visually Impaired](https://mlsanity.com/paper/2502.07183/) (2025)
- [Obstacle avoidance for blind people using a 3D camera and a haptic feedback sleeve](https://mlsanity.com/paper/2201.04453/) (2022)
- [OpenMPR: Recognize Places Using Multimodal Data for People with Visual Impairments](https://mlsanity.com/paper/1909.06795/) (2019)
- [Floor extraction and door detection for visually impaired guidance](https://mlsanity.com/paper/2401.17056/) (2024)
- [DISHA: Low-Energy Sparse Transformer at Edge for Outdoor Navigation for the Visually Impaired Individuals](https://mlsanity.com/paper/2406.15864/) (2024)

## Built on by (most cited first, 3 of 3)

- [More than the Sum: Panorama-Language Models for Adverse Omni-Scenes](https://mlsanity.com/paper/2603.09573/) (2026)
- [What if? Emulative Simulation with World Models for Situated Reasoning](https://mlsanity.com/paper/2603.06445/) (2026)
- [WalkGPT: Grounded Vision-Language Conversation with Depth-Aware Segmentation for Pedestrian Navigation](https://mlsanity.com/paper/2603.10703/) (2026)

## Abstract (arXiv)

Walking assistance in extreme or complex environments remains a significant challenge for people with blindness or low vision (BLV), largely due to the lack of a holistic scene understanding. Motivated by the real-world needs of the BLV community, we build mmWalk, a simulated multi-modal dataset that integrates multi-view sensor and accessibility-oriented features for outdoor safe navigation. Our dataset comprises 120 manually controlled, scenario-categorized walking trajectories with 62k synchronized frames. It contains over 559k panoramic images across RGB, depth, and semantic modalities. Furthermore, to emphasize real-world relevance, each trajectory involves outdoor corner cases and accessibility-specific landmarks for BLV users. Additionally, we generate mmWalkVQA, a VQA benchmark with over 69k visual question-answer triplets across 9 categories tailored for safe and informed walking assistance. We evaluate state-of-the-art Vision-Language Models (VLMs) using zero- and few-shot settings and found they struggle with our risk assessment and navigational tasks. We validate our mmWalk-finetuned model on real-world datasets and show the effectiveness of our dataset for advancing multi-modal walking assistance.