mlsanityData to 7 Oct 2026

VIAssist: Adapting Multi-modal Large Language Models for Users with Visual Impairments

arXiv 2404.02508 ↗First posted 3 Apr 2024cs.CV cs.AI cs.LGcited by 24

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Individuals with visual impairments, encompassing both partial and total difficulties in visual perception, are referred to as visually impaired (VI) people. An estimated 2.2 billion individuals worldwide are affected by visual impairments. Recent advancements in multi-modal large language models (MLLMs) have showcased their extraordinary capabilities across various domains. It is desirable to help VI individuals with MLLMs' great capabilities of visual understanding and reasoning. However, it is challenging for VI people to use MLLMs due to the difficulties in capturing the desirable images to fulfill their daily requests. For example, the target object is not fully or partially placed in the image. This paper explores how to leverage MLLMs for VI individuals to provide visual-question answers. VIAssist can identify undesired images and provide detailed actions. Finally, VIAssist can provide reliable answers to users' queries based on the images. Our results show that VIAssist provides +0.21 and +0.31 higher BERTScore and ROUGE scores than the baseline, respectively.

Key facts. Cited by 24 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 18 of its 28 references that are in the map. First posted on arXiv 3 Apr 2024.

Builds on

18 of 28 references.

References · 18

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2023Segment Anythingcited by 8,158
  5. 2023Visual Instruction Tuningcited by 7,792
  6. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  7. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  8. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  9. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  10. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  11. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  12. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 852
6 more
  1. 2024The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bitscited by 289
  2. 2024VIALM: A Survey and Benchmark of Visually Impaired Assistance with Large Modelscited by 23
  3. 2023EdgeFM: Leveraging Foundation Model for Open-set Learning on the Edgecited by 21
  4. 2023Open Scene Understanding: Grounded Situation Recognition Meets Segment Anything for Helping People with Visual Impairmentscited by 13
  5. 2019Deep Learning based Wearable Assistive System for Visually Impaired Peoplecited by 12
  6. 2022ProtoSound: A Personalized and Scalable Sound Recognition System for Deaf and Hard-of-Hearing Userscited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 15; 2026: 5.

24 papers.

Sort:

By area:

  1. 2025ContextAgent: Context-Aware Proactive LLM Agents with Open-World Sensory Perceptionscited by 44
  2. 2024DrHouse: An LLM-empowered Diagnostic Reasoning System through Harnessing Outcomes from Sensor Data and Expert Knowledgecited by 24
  3. 2024WalkVLM:Aid Visually Impaired People Walking by Vision Language Modelcited by 13
  4. 2025Beyond Visual Perception: Insights from Smartphone Interaction of Visually Impaired Users with Large Multimodal Modelscited by 13
  5. 2025Evaluating Multimodal Language Models as Visual Assistants for Visually Impaired Userscited by 11
  6. 2025GLSim: Detecting Object Hallucinations in LVLMs via Global-Local Similaritycited by 11
  7. 2024Emerging Practices for Large Multimodal Model (LMM) Assistance for People with Visual Impairments: Implications for Designcited by 10
  8. 2025ProAgent: Harnessing On-Demand Sensory Contexts for Proactive LLM Agent Systems in the Wildcited by 7
  9. 2024ObjectFinder: An Open-Vocabulary Assistive System for Interactive Object Search by Blind Peoplecited by 5
  10. 2025REVEAL: Multi-turn Evaluation of Image-Input Harms for Vision LLMcited by 4
  11. 2025A Light and Smart Wearable Platform with Multimodal Foundation Model for Enhanced Spatial Reasoning in People with Blindness and Low Visioncited by 2
  12. 2025Probing the Gaps in ChatGPT Live Video Chat for Real-World Assistance for People who are Blind or Visually Impairedcited by 2
12 more
  1. 2025Less Redundancy: Boosting Practicality of Vision Language Model in Walking Assistantscited by 2
  2. 2025NaviSense: A Multimodal Assistive Mobile application for Object Retrieval by Persons with Visual Impairmentcited by 2
  3. 2026Pro2Assist: Continuous Step-aware Proactive Assistance with Multi-modal Egocentric Perception for Long-horizon Procedural Taskscited by 2
  4. 2025An LLM-Empowered Low-Resolution Vision System for On-Device Human Behavior Understandingcited by 1
  5. 2025Multilingual Vision-Language Models, A Surveycited by 1
  6. 2025Are Large Vision-Language Models Ready to Guide Blind and Low-Vision Individuals?cited by 1
  7. 2026DEX-AR: A Dynamic Explainability Method for Autoregressive Vision-Language Modelscited by 1
  8. 2026SensorPersona: An LLM-Empowered System for Continual Persona Extraction from Longitudinal Mobile Sensor Streamscited by 1
  9. 2025EmoAssist: Emotional Assistant for Visual Impairment Communitycited by 0
  10. 2025How Blind and Low-Vision Individuals Prefer Large Vision-Language Model-Generated Scene Descriptionscited by 0
  11. 2026VIABench: A Comprehensive Video Benchmark Collected from Blind Individuals for Visual Impairment Assistancecited by 0
  12. 2026Time-Aware Assistive Navigationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Guiding Multimodal Large Language Models with Blind and Low Vision People Visual Questions for Proactive Visual Interpretations
  2. 2026How Multimodal Large Language Models Support Access to Visual Information: A Diary Study With Blind and Low Vision People
  3. 2025Towards Understanding the Use of MLLM-Enabled Applications for Visual Interpretation by Blind and Low Vision People
  4. 2025MLLMs Know Where to Look: Training-free Perception of Small Visual Details with Multimodal LLMs
  5. 2024Exploring the Frontier of Vision-Language Models: A Survey of Current Methodologies and Future Directions
  6. 2026Are We There Yet? Exploring the Capabilities of MLLMs in Assistive AI Applications
  7. 2024Efficient Multimodal Large Language Models: A Survey
  8. 2024A Survey on Benchmarks of Multimodal Large Language Models