mlsanityData to 14 Oct 2026

Focus Where It Counts: A Salience-Driven Vision-Language Model for Low Vision Assistance

arXiv 2608.28218 ↗First posted 28 Aug 2026cs.CVcited by 0

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Vision-language models (VLMs) are rapidly progressing and offer promising capabilities for assistive technologies supporting persons with blindness or low vision. However, existing VLMs are primarily designed for general-purpose captioning and do not explicitly model human perceptual priorities, thereby limiting their ability to emphasize the most relevant information in a scene. To address this gap, we propose a salience-driven captioning framework that prioritizes scene elements according to their importance for human-centered assistance. We curate three salience-aware datasets, namely, Salience COCO, Salience Flickr, and Salience VizWiz, with object-level salience annotations designed to reflect the visual information most relevant to low vision users across different environments. Building on these datasets, we introduce Salience-LLaVA, a salience-aware VLM that incorporates salience cues to generate captions in which important elements are mentioned in the order of importance. Our work makes four main contributions. We build salience-aware datasets verified by low vision participants, propose Salience-LLaVA to describe objects in the order of importance, introduce SCMI to evaluate ordering accuracy, and deploy the system on assistive glasses to demonstrate real-world practicality. Code and datasets are available at: https://github.com/topo-focus/Topofocus

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 38 of its 60 references that are in the map. First posted on arXiv 28 Aug 2026.

Builds on

38 of 60 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024WalkVLM:Aid Visually Impaired People Walking by Vision Language Model30 Dec
  2. 2023Tag2Text: Guiding Vision-Language Model via Image Tagging10 Mar
  3. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation28 Jan
  4. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervision24 Aug

Other references · 34

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  5. 2023Visual Instruction Tuningcited by 7,869
  6. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  7. 2015You Only Look Once: Unified, Real-Time Object Detectioncited by 4,852
  8. 2021Masked-attention Mask Transformer for Universal Image Segmentationcited by 2,351
  9. 2025OpenAI GPT-5 System Cardcited by 989
  10. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 858
  11. 2025Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scalingcited by 812
  12. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 745
22 more
  1. 2024YOLO-World: Real-Time Open-Vocabulary Object Detectioncited by 503
  2. 2022mPLUG: Effective and Efficient Vision-Language Learning by Cross-modal Skip-connectionscited by 187
  3. 2022From Images to Textual Prompts: Zero-shot VQA with Frozen Large Language Modelscited by 115
  4. 2020Captioning Images Taken by People Who Are Blindcited by 110
  5. 2023DeCap: Decoding CLIP Latents for Zero-Shot Captioning via Text-Only Trainingcited by 84
  6. 2022GRIT: Faster and Better Image captioning Transformer Using Dual Visual Featurescited by 56
  7. 2025UniME-V2: MLLM-as-a-Judge for Universal Multimodal Embedding Learningcited by 42
  8. 2021Injecting Semantic Concepts into End-to-End Image Captioningcited by 41
  9. 2024RAR: Retrieving And Ranking Augmented MLLMs for Visual Recognitioncited by 24
  10. 2025SCOPE: Saliency-Coverage Oriented Token Pruning for Efficient Multimodel LLMscited by 24
  11. 2024Zero-shot Object Navigation with Vision-Language Models Reasoningcited by 19
  12. 2023Sketch2Saliency: Learning to Detect Salient Objects from Human Drawingscited by 17
  13. 2024GazeXplain: Learning to Predict Natural Language Explanations of Visual Scanpathscited by 15
  14. 2017Paying Attention to Descriptions Generated by Image Captioning Modelscited by 12
  15. 2024ViSaRL: Visual Reinforcement Learning Guided by Human Saliencycited by 12
  16. 2025FedRSClip: Federated Learning for Remote Sensing Scene Classification Using Vision-Language Modelscited by 11
  17. 2023Salient Object Detection for Images Taken by People With Vision Impairmentscited by 7
  18. 2024FedMLLM: Federated Fine-tuning MLLM on Multimodal Heterogeneity Datacited by 7
  19. 2021Where do Models go Wrong? Parameter-Space Saliency Maps for Explainabilitycited by 6
  20. 2023What Do Deep Saliency Models Learn about Visual Attention?cited by 4
  21. 2024AI-based Wearable Vision Assistance System for the Visually Impaired: Integrating Real-Time Object Recognition and Contextual Understanding Using Large Vision-Language Modelscited by 2
  22. 2025Situat3DChange: Situated 3D Change Understanding Dataset for Multimodal Large Language Modelcited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024@Bench: Benchmarking Vision-Language Models for Human-centered Assistive Technology
  2. 2025Are Large Vision-Language Models Ready to Guide Blind and Low-Vision Individuals?
  3. 2025How Blind and Low-Vision Individuals Prefer Large Vision-Language Model-Generated Scene Descriptions
  4. 2026Small yet Assistive: Spatially-Aware Post-Training for Low Vision
  5. 2025Vision-Language Models Can't See the Obvious
  6. 2023Explaining CLIP's performance disparities on data from blind/low vision users
  7. 2024Vision-Language Models under Cultural and Inclusive Considerations
  8. 2025"It's trained by non-disabled people": Evaluating How Image Quality Affects Product Captioning with Vision-Language Models