# Emerging Practices for Large Multimodal Model (LMM) Assistance for People with Visual Impairments: Implications for Design

arXiv: https://arxiv.org/abs/2407.08882 · HTML: https://mlsanity.com/paper/2407.08882/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 6 of its 77 references that are in the map.
- First posted on arXiv 11 Jul 2024.

## Placed in

- Multimodal LLMs / vision-language models / [VLM benchmarks and visual reasoning](https://mlsanity.com/mllm/vlm-benchmarks-and-visual-reasoning/)

## Builds on (most cited first, 6 of 6)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities](https://mlsanity.com/paper/2308.02490/) (2023)
- [VIAssist: Adapting Multi-modal Large Language Models for Users with Visual Impairments](https://mlsanity.com/paper/2404.02508/) (2024)
- [VIALM: A Survey and Benchmark of Visually Impaired Assistance with Large Models](https://mlsanity.com/paper/2402.01735/) (2024)
- [Scene Text Detection via Holistic, Multi-Channel Prediction](https://mlsanity.com/paper/1606.09002/) (2016)
- [How Can Generative AI Enhance the Well-being of Blind?](https://mlsanity.com/paper/2402.07919/) (2024)

## Built on by (most cited first, 10 of 10)

- [Personalized Multimodal Large Language Models: A Survey](https://mlsanity.com/paper/2412.02142/) (2024)
- [WalkVLM:Aid Visually Impaired People Walking by Vision Language Model](https://mlsanity.com/paper/2412.20903/) (2024)
- [ObjectFinder: An Open-Vocabulary Assistive System for Interactive Object Search by Blind People](https://mlsanity.com/paper/2412.03118/) (2024)
- [How Multimodal Large Language Models Support Access to Visual Information: A Diary Study With Blind and Low Vision People](https://mlsanity.com/paper/2602.13469/) (2026)
- [Say It My Way: Exploring Control in Conversational Visual Question Answering with Blind Users](https://mlsanity.com/paper/2602.16930/) (2026)
- [Accessibility Considerations in the Development of an AI Action Plan](https://mlsanity.com/paper/2503.14522/) (2025)
- ["Accessibility people, you go work on that thing of yours over there": Addressing Disability Inclusion in AI Product Organizations](https://mlsanity.com/paper/2508.16607/) (2025)
- [Continual-learning for Modelling Low-Resource Languages from Large Language Models](https://mlsanity.com/paper/2601.05874/) (2026)
- [IKIWISI: An Interactive Visual Pattern Generator for Evaluating the Reliability of Vision-Language Models Without Ground Truth](https://mlsanity.com/paper/2505.22305/) (2025)
- [VIVA+: Human-Centered Situational Decision-Making](https://mlsanity.com/paper/2509.23698/) (2025)

## Abstract (arXiv)

People with visual impairments perceive their environment non-visually and often use AI-powered assistive tools to obtain textual descriptions of visual information. Recent large vision-language model-based AI-powered tools like Be My AI are more capable of understanding users' inquiries in natural language and describing the scene in audible text; however, the extent to which these tools are useful to visually impaired users is currently understudied. This paper aims to fill this gap. Our study with 14 visually impaired users reveals that they are adapting these tools organically -- not only can these tools facilitate complex interactions in household, spatial, and social contexts, but they also act as an extension of users' cognition, as if the cognition were distributed in the visual information. We also found that although the tools are currently not goal-oriented, users accommodate this limitation and embrace the tools' capabilities for broader use. These findings enable us to envision design implications for creating more goal-oriented, real-time processing, and reliable AI-powered assistive technology.