mlsanityData to 7 Oct 2026

WalkVLM:Aid Visually Impaired People Walking by Vision Language Model

arXiv 2412.20903 ↗First posted 30 Dec 2024cs.CV cs.AIcited by 13

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Approximately 200 million individuals around the world suffer from varying degrees of visual impairment, making it crucial to leverage AI technology to offer walking assistance for these people. With the recent progress of vision-language models (VLMs), applying VLMs to offer walking guidance has become popular. However, the existing methods of walking guidance are mainly based on self-curated question-answering datasets that are not publicly accessible, without a standardized benchmark for training or evaluation. Moreover, walking assistance often requires real-time streaming video analysis and the generation of concise yet informative reminders, making VLMs struggle due to excessive responses and low efficiency in inferences. In this paper, we introduce the first large-scale dataset dedicated to walking assistance, comprising 12,000 video-annotation pairs, to provide a unified benchmark for training and evaluating systems to help visually-impaired individuals walk. Furthermore, a WalkVLM model is proposed, which employs chain of thought for hierarchical planning to generate concise but informative reminders and utilizes temporal-aware adaptive prediction to reduce the temporal redundancy of reminders. Finally, we have established a solid benchmark for blind walking task and verified the advantages of WalkVLM in stream video processing for this task compared to other VLMs. Our dataset and code are available at https://walkvlm2024.github.io.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 39 of its 55 references that are in the map. First posted on arXiv 30 Dec 2024.

Builds on

39 of 55 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024Emerging Practices for Large Multimodal Model (LMM) Assistance for People with Visual Impairments: Implications for Design11 Jul
  2. 2024Math-LLaVA: Bootstrapping Mathematical Reasoning for Multimodal Large Language Models25 Jun
  3. 2024An Introduction to Vision-Language Modeling27 May
  4. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond24 Aug
  5. 2023Vision-Language Models for Vision Tasks: A Survey3 Apr
  6. 2022A Survey of Vision-Language Pre-Trained Models18 Feb

Other references · 33

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  6. 2023Segment Anythingcited by 8,158
  7. 2023Visual Instruction Tuningcited by 7,792
  8. 2024GPT-4o System Cardcited by 4,467
  9. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  10. 2022A ConvNet for the 2020scited by 3,545
  11. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  12. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,185
21 more
  1. 2024Qwen2 Technical Reportcited by 2,076
  2. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,287
  3. 2019LVIS: A Dataset for Large Vocabulary Instance Segmentationcited by 1,215
  4. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,061
  5. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 852
  6. 2024DeepSeek-VL: Towards Real-World Vision-Language Understandingcited by 718
  7. 2024Yi: Open Foundation Models by 01.AIcited by 706
  8. 2022Detecting Twenty-thousand Classes using Image-level Supervisioncited by 613
  9. 2024DriveVLM: The Convergence of Autonomous Driving and Large Vision-Language Modelscited by 482
  10. 2024SparseVLM: Visual Token Sparsification for Efficient Vision-Language Model Inferencecited by 378
  11. 2024LLaVA-Phi: Efficient Multi-Modal Assistant with Small Language Modelcited by 128
  12. 2024Efficient Multimodal Large Language Models: A Surveycited by 95
  13. 2024LLaVA-MoLE: Sparse Mixture of LoRA Experts for Mitigating Data Conflicts in Instruction Finetuning MLLMscited by 84
  14. 2024VIAssist: Adapting Multi-modal Large Language Models for Users with Visual Impairmentscited by 24
  15. 2024VIALM: A Survey and Benchmark of Visually Impaired Assistance with Large Modelscited by 23
  16. 2023Open Scene Understanding: Grounded Situation Recognition Meets Segment Anything for Helping People with Visual Impairmentscited by 13
  17. 2024VisionGPT: LLM-Assisted Real-Time Anomaly Detection for Safe Visual Navigationcited by 13
  18. 2024Generating Contextually-Relevant Navigation Instructions for Blind and Low Vision Peoplecited by 7
  19. 2024Identifying Crucial Objects in Blind and Low-Vision Individuals' Navigationcited by 5
  20. 2024A Dataset for Crucial Object Recognition in Blind and Low-Vision Individuals' Navigationcited by 4
  21. 2022"I Want to Figure Things Out": Supporting Exploration in Navigation for People with Visual Impairmentscited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 9; 2026: 4.

13 papers.

Sort:

By area:

  1. 2025A Survey of Multimodal Hallucination Evaluation and Detectioncited by 26
  2. 2025Hallucination at a Glance: Controlled Visual Edits and Fine-Grained Multimodal Learningcited by 11
  3. 2025Safe-LLaVA: A Privacy-Preserving Vision-Language Dataset and Benchmark for Biometric Safetycited by 7
  4. 2025Poivre: Self-Refining Visual Pointing with Reinforcement Learningcited by 3
  5. 2025Less Redundancy: Boosting Practicality of Vision Language Model in Walking Assistantscited by 2
  6. 2026WalkGPT: Grounded Vision-Language Conversation with Depth-Aware Segmentation for Pedestrian Navigationcited by 2
  7. 2025Are Large Vision-Language Models Ready to Guide Blind and Low-Vision Individuals?cited by 1
  8. 2025How Blind and Low-Vision Individuals Prefer Large Vision-Language Model-Generated Scene Descriptionscited by 0
  9. 2025FoCTTA: Low-Memory Continual Test-Time Adaptation with Focuscited by 0
  10. 2025A Synthetic-to-Real Dehazing Method based on Domain Unificationcited by 0
  11. 2026Urban Risk-Aware Navigation via VQA-Based Event Maps for People with Low Visioncited by 0
  12. 2026VIABench: A Comprehensive Video Benchmark Collected from Blind Individuals for Visual Impairment Assistancecited by 0
1 more
  1. 2026Focus Where It Counts: A Salience-Driven Vision-Language Model for Low Vision Assistancecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025mmWalk: Towards Multi-modal Multi-view Walking Assistance
  2. 2026Exploring the Use of VLMs for Navigation Assistance for People with Blindness and Low Vision
  3. 2024@Bench: Benchmarking Vision-Language Models for Human-centered Assistive Technology
  4. 2025GuideDog: A Real-World Egocentric Multimodal Dataset for Blind and Low-Vision Accessibility-Aware Guidance
  5. 2026Assisted Spatial Cognition Through Vision-Language Models
  6. 2026Small yet Assistive: Spatially-Aware Post-Training for Low Vision
  7. 2026Time-Aware Assistive Navigation
  8. 2025Towards Blind and Low-Vision Accessibility of Lightweight VLMs and Custom LLM-Evals