mlsanityData to 14 Oct 2026

Beyond Visual Perception: Insights from Smartphone Interaction of Visually Impaired Users with Large Multimodal Models

arXiv 2502.16098 ↗First posted 22 Feb 2025cs.HCcited by 13

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Large multimodal models (LMMs) have enabled new AI-powered applications that help people with visual impairments (PVI) receive natural language descriptions of their surroundings through audible text. We investigated how this emerging paradigm of visual assistance transforms how PVI perform and manage their daily tasks. Moving beyond usability assessments, we examined both the capabilities and limitations of LMM-based tools in personal and social contexts, while exploring design implications for their future development. Through interviews with 14 visually impaired users of Be My AI (an LMM-based application) and analysis of its image descriptions from both study participants and social media platforms, we identified two key limitations. First, these systems' context awareness suffers from hallucinations and misinterpretations of social contexts, styles, and human identities. Second, their intent-oriented capabilities often fail to grasp and act on users' intentions. Based on these findings, we propose design strategies for improving both human-AI and AI-AI interactions, contributing to the development of more effective, interactive, and personalized assistive technologies.

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 23 of its 67 references that are in the map. First posted on arXiv 22 Feb 2025.

Builds on

23 of 67 references.

References · 23

  1. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  2. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  3. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  4. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,239
  5. 2022Red Teaming Language Models with Language Modelscited by 1,026
  6. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 858
  7. 2023MemoryBank: Enhancing Large Language Models with Long-Term Memorycited by 556
  8. 2020Consistent Estimators for Learning to Defer to an Expertcited by 271
  9. 2019Towards Fairness in Visual Recognition: Effective Strategies for Bias Mitigationcited by 252
  10. 2020Fair Attribute Classification through Latent Space De-biasingcited by 125
  11. 2020Captioning Images Taken by People Who Are Blindcited by 110
  12. 2019The Algorithmic Automation Problem: Prediction, Triage, and Human Effortcited by 101
11 more
  1. 2024VIAssist: Adapting Multi-modal Large Language Models for Users with Visual Impairmentscited by 25
  2. 2024Investigating Use Cases of AI-Powered Scene Description Applications for Blind and Low Vision Peoplecited by 24
  3. 2024VIALM: A Survey and Benchmark of Visually Impaired Assistance with Large Modelscited by 23
  4. 2021Computer Vision and Conflicting Values: Describing People with Automated Alt Textcited by 14
  5. 2024Enhancing the Travel Experience for People with Visual Impairments through Multimodal Interaction: NaviGPT, A Real-Time AI-Driven Mobile Navigation Systemcited by 13
  6. 2024"It's Kind of Context Dependent": Understanding Blind and Low Vision People's Video Accessibility Preferences Across Viewing Scenarioscited by 11
  7. 2018Conversations for Vision: Remote Sighted Assistants Helping People with Visual Impairmentscited by 6
  8. 2022Blind Users Accessing Their Training Images in Teachable Object Recognizerscited by 6
  9. 2024How Can Generative AI Enhance the Well-being of Blind?cited by 5
  10. 2024Audio Description Customizationcited by 4
  11. 2024Unblind Text Inputs: Predicting Hint-text of Text Input in Mobile Apps via LLMcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 6; 2026: 7.

13 papers.

Sort:

By area:

  1. 2025"Before, I Asked My Mom, Now I Ask ChatGPT": Visual Privacy Management with Generative AI for Blind and Low-Vision Peoplecited by 6
  2. 2025"I Can See Forever!": Evaluating Real-time VideoLLMs for Assisting Individuals with Visual Impairmentscited by 3
  3. 2025"It's trained by non-disabled people": Evaluating How Image Quality Affects Product Captioning with Vision-Language Modelscited by 3
  4. 2026How Multimodal Large Language Models Support Access to Visual Information: A Diary Study With Blind and Low Vision Peoplecited by 3
  5. 2025Probing the Gaps in ChatGPT Live Video Chat for Real-World Assistance for People who are Blind or Visually Impairedcited by 2
  6. 2025IKIWISI: An Interactive Visual Pattern Generator for Evaluating the Reliability of Vision-Language Models Without Ground Truthcited by 0
  7. 2025Guiding Multimodal Large Language Models with Blind and Low Vision People Visual Questions for Proactive Visual Interpretationscited by 0
  8. 2026RPIQ: Residual-Projected Multi-Collaboration Closed-Loop and Single Instance Quantization for Visually Impaired Assistancecited by 0
  9. 2026Auditing Disability Representation in Vision-Language Modelscited by 0
  10. 2026Lost in Instructions: Study of Blind Users' Experiences with DIY Manuals and AI-Rewritten Instructions for Assembly, Operation, and Troubleshooting of Tangible Productscited by 0
  11. 2026Bespoke Visual Assistance: What and How do Blind and Low-Vision People Create with Agentic Programming?cited by 0
  12. 2026More Than Just Access: Generative AI as Communication Intermediary for Blind and Low-Vision Userscited by 0
1 more
  1. 2026Touvigation: Embodied Adaptive Object Acquisition for Blind and Low-Vision Users in Unfamiliar Indoor Environmentscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Emerging Practices for Large Multimodal Model (LMM) Assistance for People with Visual Impairments: Implications for Design
  2. 2025Evaluating Multimodal Language Models as Visual Assistants for Visually Impaired Users
  3. 2026From Gaze to Guidance: Interpreting and Adapting to Users' Cognitive Needs with Multimodal Gaze-Aware AI Assistants
  4. 2025NaviSense: A Multimodal Assistive Mobile application for Object Retrieval by Persons with Visual Impairment
  5. 2026VisionAId: An Offline-First Multimodal Android Assistant for People with Visual Impairment, Featuring Personalized Object Retrieval
  6. 2025Audo-Sight: Enabling Ambient Interaction For Blind And Visually Impaired Individuals
  7. 2024Human I/O: Towards a Unified Approach to Detecting Situational Impairments
  8. 2025Towards Understanding the Use of MLLM-Enabled Applications for Visual Interpretation by Blind and Low Vision People