mlsanityData to 14 Oct 2026

Evaluating Multimodal Language Models as Visual Assistants for Visually Impaired Users

arXiv 2503.22610 ↗First posted 28 Mar 2025cs.HC cs.AI cs.CLcited by 11

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

This paper explores the effectiveness of Multimodal Large Language models (MLLMs) as assistive technologies for visually impaired individuals. We conduct a user survey to identify adoption patterns and key challenges users face with such technologies. Despite a high adoption rate of these models, our findings highlight concerns related to contextual understanding, cultural sensitivity, and complex scene understanding, particularly for individuals who may rely solely on them for visual interpretation. Informed by these results, we collate five user-centred tasks with image and video inputs, including a novel task on Optical Braille Recognition. Our systematic evaluation of twelve MLLMs reveals that further advancements are necessary to overcome limitations related to cultural context, multilingual support, Braille reading comprehension, assistive object recognition, and hallucinations. This work provides critical insights into the future direction of multimodal AI for accessibility, underscoring the need for more inclusive, robust, and trustworthy visual assistance technologies.

Key facts. Cited by 11 papers in the map (data through 14 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 53 of its 90 references that are in the map. First posted on arXiv 28 Mar 2025.

Builds on

53 of 90 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024Long-Form Answers to Visual Questions from Blind and Low Vision People12 Aug
  2. 2024Benchmarking Vision Language Models for Cultural Understanding15 Jul
  3. 2024Vision-Language Models under Cultural and Inclusive Considerations8 Jul
  4. 2024African or European Swallow? Benchmarking Large Vision-Language Models for Fine-Grained Object Classification20 Jun
  5. 2024Measuring Multimodal Mathematical Reasoning with MATH-Vision Dataset22 Feb
  6. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts3 Oct

Other references · 47

  1. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  2. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  3. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  4. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  5. 2015VQA: Visual Question Answeringcited by 2,506
  6. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,424
  7. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,917
  8. 2017The Kinetics Human Action Video Datasetcited by 1,909
  9. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,556
  10. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,554
  11. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,312
  12. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,109
35 more
  1. 2017The "something something" video database for learning and evaluating visual common sensecited by 1,103
  2. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,079
  3. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 858
  4. 2020The State and Fate of Linguistic Diversity and Inclusion in the NLP Worldcited by 798
  5. 2018Scaling Egocentric Vision: The EPIC-KITCHENS Datasetcited by 795
  6. 2021NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actionscited by 744
  7. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMscited by 735
  8. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 710
  9. 2023EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understandingcited by 680
  10. 2019ActivityNet-QA: A Dataset for Understanding Complex Web Videos via Question Answeringcited by 561
  11. 2023Perception Test: A Diagnostic Benchmark for Multimodal Video Modelscited by 314
  12. 2017CORe50: a New Dataset and Benchmark for Continuous Object Recognitioncited by 289
  13. 2024Enhancing the Reasoning Ability of Multimodal Large Language Models via Mixed Preference Optimizationcited by 239
  14. 2024VideoChat-Flash: Hierarchical Compression for Long-Context Video Modelingcited by 220
  15. 2024SEED-Bench-2-Plus: Benchmarking Multimodal Large Language Models with Text-Rich Visual Comprehensioncited by 146
  16. 2023Open-domain Visual Entity Recognition: Towards Recognizing Millions of Wikipedia Entitiescited by 113
  17. 2020Participation is not a Design Fix for Machine Learningcited by 111
  18. 2020Captioning Images Taken by People Who Are Blindcited by 110
  19. 2022Crossmodal-3600: A Massively Multilingual Multimodal Evaluation Datasetcited by 100
  20. 2021xGQA: Cross-Lingual Visual Question Answeringcited by 72
  21. 2024Pangea: A Fully Open Multilingual Multimodal LLM for 39 Languagescited by 64
  22. 2023Rethinking Model Evaluation as Narrowing the Socio-Technical Gapcited by 56
  23. 2023The Past, Present and Better Future of Feedback Learning in Large Language Models for Subjective Human Preferences and Valuescited by 54
  24. 2024LLMs as Research Tools: A Large Scale Survey of Researchers' Usage and Perceptionscited by 47
  25. 2021Webly Supervised Fine-Grained Recognition: Benchmark Datasets and An Approachcited by 32
  26. 2021ORBIT: A Real-World Few-Shot Dataset for Teachable Object Recognitioncited by 29
  27. 2024VIAssist: Adapting Multi-modal Large Language Models for Users with Visual Impairmentscited by 25
  28. 2020Image Captioning as an Assistive Technology: Lessons Learned from VizWiz 2020 Challengecited by 17
  29. 2023VQA Therapy: Exploring Answer Differences by Visually Grounding Answerscited by 14
  30. 2024Right this way: Can VLMs Guide Us to See More to Answer Questions?cited by 10
  31. 2021Multi-Modal Image Captioning for the Visually Impairedcited by 8
  32. 2023Quality-agnostic Image Captioning to Safely Assist People with Vision Impairmentcited by 8
  33. 2023Salient Object Detection for Images Taken by People With Vision Impairmentscited by 7
  34. 2024Identifying Crucial Objects in Blind and Low-Vision Individuals' Navigationcited by 6
  35. 2023Towards Real Time Egocentric Segment Captioning for The Blind and Visually Impaired in RGB-D Theatre Imagescited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 4; 2026: 7.

11 papers.

  1. 2025RAVENEA: A Benchmark for Multimodal Retrieval-Augmented Visual Culture Understandingcited by 9
  2. 2026GlotOCR Bench: OCR Models Still Struggle Beyond a Handful of Unicode Scriptscited by 2
  3. 2025MATE: LLM-Powered Multi-Agent Translation Environment for Accessibility Applicationscited by 1
  4. 2025User Perceptions vs. Proxy LLM Judges: Privacy and Helpfulness in LLM Responses to Privacy-Sensitive Scenarioscited by 1
  5. 2025Think-Reflect-Revise: A Policy-Guided Reflective Framework for Safety Alignment in Large Vision Language Modelscited by 1
  6. 2026Are We There Yet? Exploring the Capabilities of MLLMs in Assistive AI Applicationscited by 1
  7. 2026One Framework for All: Cross-Modal Membership Inference for Generative Modelscited by 0
  8. 2026MAP: A Benchmark on Multimodal Accessibility Planning for Real World Placescited by 0
  9. 2026PuzzleMate: Benchmarking MLLMs for Egocentric Puzzle Assistancecited by 0
  10. 2026Visual Input and Its Framing Affect Attribute-based Descriptions Produced by Large Vision-Language Modelscited by 0
  11. 2026CultureMINE: Datasets and Methods for Improving the Cultural Capabilities of NLP Systemscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026How Multimodal Large Language Models Support Access to Visual Information: A Diary Study With Blind and Low Vision People
  2. 2024Emerging Practices for Large Multimodal Model (LMM) Assistance for People with Visual Impairments: Implications for Design
  3. 2025Beyond Visual Perception: Insights from Smartphone Interaction of Visually Impaired Users with Large Multimodal Models
  4. 2025Towards Understanding the Use of MLLM-Enabled Applications for Visual Interpretation by Blind and Low Vision People
  5. 2025Guiding Multimodal Large Language Models with Blind and Low Vision People Visual Questions for Proactive Visual Interpretations
  6. 2025A Light and Smart Wearable Platform with Multimodal Foundation Model for Enhanced Spatial Reasoning in People with Blindness and Low Vision
  7. 2026Assisted Spatial Cognition Through Vision-Language Models
  8. 2026Exploring the Use of VLMs for Navigation Assistance for People with Blindness and Low Vision