mlsanityData to 7 Oct 2026

Guiding Multimodal Large Language Models with Blind and Low Vision People Visual Questions for Proactive Visual Interpretations

arXiv 2510.01576 ↗First posted 2 Oct 2025cs.CV cs.AI cs.HCcited by 0

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Multimodal large language models (MLLMs) have been integrated into visual interpretation applications to support Blind and Low Vision (BLV) users because of their accuracy and ability to provide rich, human-like interpretations. However, these applications often default to comprehensive, lengthy descriptions regardless of context. This leads to inefficient exchanges, as users must go through irrelevant details rather than receiving the specific information they are likely to seek. To deliver more contextually-relevant information, we developed a system that draws on historical BLV users questions. When given an image, our system identifies similar past visual contexts from the VizWiz-LF dataset and uses the associated questions to guide the MLLM generate descriptions more relevant to BLV users. An evaluation with three human labelers who revised 92 context-aware and context-free descriptions showed that context-aware descriptions anticipated and answered users' questions in 76.1% of cases (70 out of 92) and were preferred in 54.4% of comparisons (50 out of 92). Our paper reviews, and data analysis are publicly available in a Github repository at https://github.com/rgonzalezp/guiding-multimodal-large-language-models-with-blind-and-low-vision-people-visual-questions .

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 6 of its 14 references that are in the map. First posted on arXiv 2 Oct 2025.

Builds on

6 of 14 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025Beyond Visual Perception: Insights from Smartphone Interaction of Visually Impaired Users with Large Multimodal Models22 Feb
  2. 2024Long-Form Answers to Visual Questions from Blind and Low Vision People12 Aug

Other references · 4

  1. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 852
  2. 2024Investigating Use Cases of AI-Powered Scene Description Applications for Blind and Low Vision Peoplecited by 24
  3. 2025Towards Understanding the Use of MLLM-Enabled Applications for Visual Interpretation by Blind and Low Vision Peoplecited by 11
  4. 2024Context-Aware Image Descriptions for Web Accessibilitycited by 7

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026How Multimodal Large Language Models Support Access to Visual Information: A Diary Study With Blind and Low Vision People
  2. 2025Are Large Vision-Language Models Ready to Guide Blind and Low-Vision Individuals?
  3. 2024Explainable and Interpretable Multimodal Large Language Models: A Comprehensive Survey
  4. 2025MLLMs Know Where to Look: Training-free Perception of Small Visual Details with Multimodal LLMs
  5. 2025How Blind and Low-Vision Individuals Prefer Large Vision-Language Model-Generated Scene Descriptions
  6. 2023Q-Bench: A Benchmark for General-Purpose Foundation Models on Low-level Vision
  7. 2025Evaluating Multimodal Language Models as Visual Assistants for Visually Impaired Users
  8. 2024Exploring the Frontier of Vision-Language Models: A Survey of Current Methodologies and Future Directions