A Large Vision-Language Model based Environment Perception System for Visually Impaired People
On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning
Abstract (arXiv)
It is a challenging task for visually impaired people to perceive their surrounding environment due to the complexity of the natural scenes. Their personal and social activities are thus highly limited. This paper introduces a Large Vision-Language Model(LVLM) based environment perception system which helps them to better understand the surrounding environment, by capturing the current scene they face with a wearable device, and then letting them retrieve the analysis results through the device. The visually impaired people could acquire a global description of the scene by long pressing the screen to activate the LVLM output, retrieve the categories of the objects in the scene resulting from a segmentation model by tapping or swiping the screen, and get a detailed description of the objects they are interested in by double-tapping the screen. To help visually impaired people more accurately perceive the world, this paper proposes incorporating the segmentation result of the RGB image as external knowledge into the input of LVLM to reduce the LVLM's hallucination. Technical experiments on POPE, MME and LLaVA-QA90 show that the system could provide a more accurate description of the scene compared to Qwen-VL-Chat, exploratory experiments show that the system helps visually impaired people to perceive the surrounding environment effectively.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 34 of its 57 references that are in the map. First posted on arXiv 25 Apr 2025.
Builds on
34 of 57 references.
Closest ancestors · 3
Same direction, newest first
- 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond24 Aug
- 2022Flamingo: a Visual Language Model for Few-Shot Learning29 Apr
- 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervision24 Aug
Other references · 31
- 2017Attention Is All You Needcited by 60,361
- 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
- 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
- 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,639
- 2023Visual Instruction Tuningcited by 7,869
- 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
- 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,859
- 2015You Only Look Once: Unified, Real-Time Object Detectioncited by 4,852
- 2016DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFscited by 3,241
- 2015SSD: Single Shot MultiBox Detectorcited by 3,130
- 2013Rich feature hierarchies for accurate object detection and semantic segmentationcited by 2,162
- 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,729
19 more
- 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,652
- 2015SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentationcited by 1,487
- 2015Show, Attend and Tell: Neural Image Caption Generation with Visual Attentioncited by 1,423
- 2020Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformerscited by 1,369
- 2014Show and Tell: A Neural Image Caption Generatorcited by 879
- 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 661
- 2023Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuningcited by 445
- 2021Unifying Vision-and-Language Tasks via Text Generationcited by 412
- 2023Check Your Facts and Try Again: Improving Large Language Models with External Knowledge and Automated Feedbackcited by 353
- 2023Detecting and Preventing Hallucinations in Large Vision Language Modelscited by 256
- 2016Knowing When to Look: Adaptive Attention via A Visual Sentinel for Image Captioningcited by 232
- 2023Woodpecker: Hallucination Correction for Multimodal Large Language Modelscited by 226
- 2018RedNet: Residual Encoder-Decoder Network for indoor RGB-D Semantic Segmentationcited by 66
- 2016Dense Captioning with Joint Inference and Visual Contextcited by 40
- 2019Incorporating External Knowledge into Machine Reading for Generative Question Answeringcited by 15
- 2017Smart Guiding Glasses for Visually Impaired People in Indoor Environmentcited by 14
- 2022VizWiz-FewShot: Locating Objects in Images Taken by People With Visual Impairmentscited by 11
- 2020Hand-Priming in Object Localization for Assistive Egocentric Visioncited by 8
- 2019Virtual-Blind-Road Following Based Wearable Navigation Device for Blind Peoplecited by 4
Built on by
Citing papers per year: 2026: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2025Are Large Vision-Language Models Ready to Guide Blind and Low-Vision Individuals?
- 2026Exploring the Use of VLMs for Navigation Assistance for People with Blindness and Low Vision
- 2023A Multi-Modal Foundation Model to Assist People with Blindness and Low Vision in Environmental Interaction
- 2025A Light and Smart Wearable Platform with Multimodal Foundation Model for Enhanced Spatial Reasoning in People with Blindness and Low Vision
- 2025Evaluating Multimodal Language Models as Visual Assistants for Visually Impaired Users
- 2025How Blind and Low-Vision Individuals Prefer Large Vision-Language Model-Generated Scene Descriptions
- 2024Mitigating Hallucinations in Large Vision-Language Models (LVLMs) via Language-Contrastive Decoding (LCD)
- 2024Vision-Language Models under Cultural and Inclusive Considerations