mlsanityData to 7 Oct 2026

VIALM: A Survey and Benchmark of Visually Impaired Assistance with Large Models

arXiv 2402.01735 ↗First posted 29 Jan 2024cs.CL cs.AI cs.CVcited by 23

Not on the map. Why?

Abstract (arXiv)

Visually Impaired Assistance (VIA) aims to automatically help the visually impaired (VI) handle daily activities. The advancement of VIA primarily depends on developments in Computer Vision (CV) and Natural Language Processing (NLP), both of which exhibit cutting-edge paradigms with large models (LMs). Furthermore, LMs have shown exceptional multimodal abilities to tackle challenging physically-grounded tasks such as embodied robots. To investigate the potential and limitations of state-of-the-art (SOTA) LMs' capabilities in VIA applications, we present an extensive study for the task of VIA with LMs (VIALM). In this task, given an image illustrating the physical environments and a linguistic request from a VI user, VIALM aims to output step-by-step guidance to assist the VI user in fulfilling the request grounded in the environment. The study consists of a survey reviewing recent LM research and benchmark experiments examining selected LMs' capabilities in VIA. The results indicate that while LMs can potentially benefit VIA, their output cannot be well environment-grounded (i.e., 25.7% GPT-4's responses) and lacks fine-grained guidance (i.e., 32.1% GPT-4's responses).

Key facts. Cited by 23 papers in the map (data through 7 Oct 2026). Builds on 57 of its 75 references that are in the map. First posted on arXiv 29 Jan 2024.

Builds on

57 of 75 references.

References · 57

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2023GPT-4 Technical Reportcited by 18,132
  5. 2022Training language models to follow instructions with human feedbackcited by 15,712
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  7. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  8. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  9. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  10. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  11. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  12. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
38 more
  1. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  2. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  3. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  4. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,410
  5. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  6. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  7. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  8. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  9. 2023A Survey of Large Language Modelscited by 2,537
  10. 2022Emergent Abilities of Large Language Modelscited by 2,483
  11. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  12. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,081
  13. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  14. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  15. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,445
  16. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,422
  17. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  18. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,172
  19. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,057
  20. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,029
  21. 2021Scaling Vision Transformerscited by 993
  22. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 890
  23. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 852
  24. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 837
  25. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 814
  26. 2023EVA-CLIP: Improved Training Techniques for CLIP at Scalecited by 786
  27. 2022Galactica: A Large Language Model for Sciencecited by 736
  28. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 704
  29. 2023Scaling Vision Transformers to 22 Billion Parameterscited by 690
  30. 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 685
  31. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 617
  32. 2022ProgPrompt: Generating Situated Robot Task Plans using Large Language Modelscited by 598
  33. 2023The Falcon Series of Open Language Modelscited by 526
  34. 2023VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Taskscited by 499
  35. 2022LLM-Planner: Few-Shot Grounded Planning for Embodied Agents with Large Language Modelscited by 459
  36. 2018VirtualHome: Simulating Household Activities via Programscited by 411
  37. 2021High-Performance Large-Scale Image Recognition Without Normalizationcited by 359
  38. 2023Ghost in the Minecraft: Generally Capable Agents for Open-World Environments via Large Language Models with Text-based Knowledge and Memorycited by 288

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 5; 2025: 13; 2026: 5.

23 papers.

Sort:

By area:

  1. 2024VIAssist: Adapting Multi-modal Large Language Models for Users with Visual Impairmentscited by 24
  2. 2024VisionGPT: LLM-Assisted Real-Time Anomaly Detection for Safe Visual Navigationcited by 13
  3. 2024WalkVLM:Aid Visually Impaired People Walking by Vision Language Modelcited by 13
  4. 2025Beyond Visual Perception: Insights from Smartphone Interaction of Visually Impaired Users with Large Multimodal Modelscited by 13
  5. 2024Emerging Practices for Large Multimodal Model (LMM) Assistance for People with Visual Impairments: Implications for Designcited by 10
  6. 2025Gaze-VLM:Bridging Gaze and VLMs through Attention Regularization for Egocentric Understandingcited by 10
  7. 2026Gaze-Regularized Vision-Language-Action Models for Robotic Manipulationcited by 7
  8. 2024ObjectFinder: An Open-Vocabulary Assistive System for Interactive Object Search by Blind Peoplecited by 5
  9. 2025Space-Aware Instruction Tuning: Dataset and Benchmark for Guide Dog Robots Assisting the Visually Impairedcited by 4
  10. 2025"I Can See Forever!": Evaluating Real-time VideoLLMs for Assisting Individuals with Visual Impairmentscited by 3
  11. 2025"It's trained by non-disabled people": Evaluating How Image Quality Affects Product Captioning with Vision-Language Modelscited by 3
  12. 2025Prompting without Panic: Attribute-aware, Zero-shot, Test-Time Calibrationcited by 2
11 more
  1. 2025Situat3DChange: Situated 3D Change Understanding Dataset for Multimodal Large Language Modelcited by 2
  2. 2025Are Large Vision-Language Models Ready to Guide Blind and Low-Vision Individuals?cited by 1
  3. 2026DEX-AR: A Dynamic Explainability Method for Autoregressive Vision-Language Modelscited by 1
  4. 2025EmoAssist: Emotional Assistant for Visual Impairment Communitycited by 0
  5. 2025How Blind and Low-Vision Individuals Prefer Large Vision-Language Model-Generated Scene Descriptionscited by 0
  6. 2025From Semantics, Scene to Instance-awareness: Distilling Foundation Model for Grounded Open-vocabulary Situation Recognitioncited by 0
  7. 2025Not There Yet: Evaluating Vision Language Models in Simulating the Visual Perception of People with Low Visioncited by 0
  8. 2025VIVA+: Human-Centered Situational Decision-Makingcited by 0
  9. 2026RPIQ: Residual-Projected Multi-Collaboration Closed-Loop and Single Instance Quantization for Visually Impaired Assistancecited by 0
  10. 2026Gaze-Regularized VLMs for Ego-Centric Behavior Understandingcited by 0
  11. 2026Urban Risk-Aware Navigation via VQA-Based Event Maps for People with Low Visioncited by 0