# VIAssist: Adapting Multi-modal Large Language Models for Users with Visual Impairments

arXiv: https://arxiv.org/abs/2404.02508 · HTML: https://mlsanity.com/paper/2404.02508/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 18 of its 28 references that are in the map.
- First posted on arXiv 3 Apr 2024.

## Placed in

- Multimodal LLMs / vision-language models / [MLLM training and visual instruction tuning](https://mlsanity.com/mllm/mllm-training-and-visual-instruction-tuning/)

## Builds on (most cited first, 18 of 18)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://mlsanity.com/paper/2205.14135/) (2022)
- [MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models](https://mlsanity.com/paper/2304.10592/) (2023)
- [Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering](https://mlsanity.com/paper/1612.00837/) (2016)
- [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://mlsanity.com/paper/2107.13586/) (2021)
- [ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision](https://mlsanity.com/paper/2102.03334/) (2021)
- [VizWiz Grand Challenge: Answering Visual Questions from Blind People](https://mlsanity.com/paper/1802.08218/) (2018)
- [The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits](https://mlsanity.com/paper/2402.17764/) (2024)
- [VIALM: A Survey and Benchmark of Visually Impaired Assistance with Large Models](https://mlsanity.com/paper/2402.01735/) (2024)
- [EdgeFM: Leveraging Foundation Model for Open-set Learning on the Edge](https://mlsanity.com/paper/2311.10986/) (2023)
- [Open Scene Understanding: Grounded Situation Recognition Meets Segment Anything for Helping People with Visual Impairments](https://mlsanity.com/paper/2307.07757/) (2023)
- [Deep Learning based Wearable Assistive System for Visually Impaired People](https://mlsanity.com/paper/1908.03364/) (2019)
- [ProtoSound: A Personalized and Scalable Sound Recognition System for Deaf and Hard-of-Hearing Users](https://mlsanity.com/paper/2202.11134/) (2022)

## Built on by (most cited first, 25 of 25)

- [ContextAgent: Context-Aware Proactive LLM Agents with Open-World Sensory Perceptions](https://mlsanity.com/paper/2505.14668/) (2025)
- [DrHouse: An LLM-empowered Diagnostic Reasoning System through Harnessing Outcomes from Sensor Data and Expert Knowledge](https://mlsanity.com/paper/2405.12541/) (2024)
- [WalkVLM:Aid Visually Impaired People Walking by Vision Language Model](https://mlsanity.com/paper/2412.20903/) (2024)
- [Beyond Visual Perception: Insights from Smartphone Interaction of Visually Impaired Users with Large Multimodal Models](https://mlsanity.com/paper/2502.16098/) (2025)
- [Evaluating Multimodal Language Models as Visual Assistants for Visually Impaired Users](https://mlsanity.com/paper/2503.22610/) (2025)
- [GLSim: Detecting Object Hallucinations in LVLMs via Global-Local Similarity](https://mlsanity.com/paper/2508.19972/) (2025)
- [Emerging Practices for Large Multimodal Model (LMM) Assistance for People with Visual Impairments: Implications for Design](https://mlsanity.com/paper/2407.08882/) (2024)
- [ProAgent: Harnessing On-Demand Sensory Contexts for Proactive LLM Agent Systems in the Wild](https://mlsanity.com/paper/2512.06721/) (2025)
- [ObjectFinder: An Open-Vocabulary Assistive System for Interactive Object Search by Blind People](https://mlsanity.com/paper/2412.03118/) (2024)
- [REVEAL: Multi-turn Evaluation of Image-Input Harms for Vision LLM](https://mlsanity.com/paper/2505.04673/) (2025)
- [Pro2Assist: Continuous Step-aware Proactive Assistance with Multi-modal Egocentric Perception for Long-horizon Procedural Tasks](https://mlsanity.com/paper/2605.04227/) (2026)
- [A Light and Smart Wearable Platform with Multimodal Foundation Model for Enhanced Spatial Reasoning in People with Blindness and Low Vision](https://mlsanity.com/paper/2505.10875/) (2025)
- [Probing the Gaps in ChatGPT Live Video Chat for Real-World Assistance for People who are Blind or Visually Impaired](https://mlsanity.com/paper/2508.03651/) (2025)
- [Less Redundancy: Boosting Practicality of Vision Language Model in Walking Assistants](https://mlsanity.com/paper/2508.16070/) (2025)
- [NaviSense: A Multimodal Assistive Mobile application for Object Retrieval by Persons with Visual Impairment](https://mlsanity.com/paper/2509.18672/) (2025)
- [An LLM-Empowered Low-Resolution Vision System for On-Device Human Behavior Understanding](https://mlsanity.com/paper/2505.01743/) (2025)
- [Multilingual Vision-Language Models, A Survey](https://mlsanity.com/paper/2509.22123/) (2025)
- [Are Large Vision-Language Models Ready to Guide Blind and Low-Vision Individuals?](https://mlsanity.com/paper/2510.00766/) (2025)
- [DEX-AR: A Dynamic Explainability Method for Autoregressive Vision-Language Models](https://mlsanity.com/paper/2603.06302/) (2026)
- [SensorPersona: An LLM-Empowered System for Continual Persona Extraction from Longitudinal Mobile Sensor Streams](https://mlsanity.com/paper/2604.06204/) (2026)
- [EmoAssist: Emotional Assistant for Visual Impairment Community](https://mlsanity.com/paper/2502.09285/) (2025)
- [How Blind and Low-Vision Individuals Prefer Large Vision-Language Model-Generated Scene Descriptions](https://mlsanity.com/paper/2502.14883/) (2025)
- [Devil in the Lens: Analyzing and Defending Physical Prompt Injection Against Vision-Language Models on Wearable Devices](https://mlsanity.com/paper/2607.10269/) (2026)
- [VIABench: A Comprehensive Video Benchmark Collected from Blind Individuals for Visual Impairment Assistance](https://mlsanity.com/paper/2607.14660/) (2026)
- [Time-Aware Assistive Navigation](https://mlsanity.com/paper/2609.05596/) (2026)

## Abstract (arXiv)

Individuals with visual impairments, encompassing both partial and total difficulties in visual perception, are referred to as visually impaired (VI) people. An estimated 2.2 billion individuals worldwide are affected by visual impairments. Recent advancements in multi-modal large language models (MLLMs) have showcased their extraordinary capabilities across various domains. It is desirable to help VI individuals with MLLMs' great capabilities of visual understanding and reasoning. However, it is challenging for VI people to use MLLMs due to the difficulties in capturing the desirable images to fulfill their daily requests. For example, the target object is not fully or partially placed in the image. This paper explores how to leverage MLLMs for VI individuals to provide visual-question answers. VIAssist can identify undesired images and provide detailed actions. Finally, VIAssist can provide reliable answers to users' queries based on the images. Our results show that VIAssist provides +0.21 and +0.31 higher BERTScore and ROUGE scores than the baseline, respectively.