mlsanityData to 7 Oct 2026

@Bench: Benchmarking Vision-Language Models for Human-centered Assistive Technology

arXiv 2409.14215 ↗First posted 21 Sep 2024cs.CVcited by 2

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

As Vision-Language Models (VLMs) advance, human-centered Assistive Technologies (ATs) for helping People with Visual Impairments (PVIs) are evolving into generalists, capable of performing multiple tasks simultaneously. However, benchmarking VLMs for ATs remains under-explored. To bridge this gap, we first create a novel AT benchmark (@Bench). Guided by a pre-design user study with PVIs, our benchmark includes the five most crucial vision-language tasks: Panoptic Segmentation, Depth Estimation, Optical Character Recognition (OCR), Image Captioning, and Visual Question Answering (VQA). Besides, we propose a novel AT model (@Model) that addresses all tasks simultaneously and can be expanded to more assistive functions for helping PVIs. Our framework exhibits outstanding performance across tasks by integrating multi-modal information, and it offers PVIs a more comprehensive assistance. Extensive experiments prove the effectiveness and generalizability of our framework.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 37 of its 56 references that are in the map. First posted on arXiv 21 Sep 2024.

Builds on

37 of 56 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022VLUE: A Multi-Task Benchmark for Evaluating Vision-Language Models30 May

Other references · 36

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020End-to-End Object Detection with Transformerscited by 7,384
  3. 2021Masked-attention Mask Transformer for Universal Image Segmentationcited by 2,338
  4. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  5. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  6. 2021Vision Transformers for Dense Predictioncited by 1,789
  7. 2014Depth Map Prediction from a Single Image using a Multi-Scale Deep Networkcited by 1,264
  8. 2021Per-Pixel Classification is Not All You Need for Semantic Segmentationcited by 1,148
  9. 2016Semantic Understanding of Scenes through the ADE20K Datasetcited by 1,063
  10. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 852
  11. 2018Panoptic Segmentationcited by 796
  12. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 762
24 more
  1. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
  2. 2023AGIEval: A Human-Centric Benchmark for Evaluating Foundation Modelscited by 701
  3. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 536
  4. 2022Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Taskscited by 420
  5. 2019From Big to Small: Multi-Scale Local Planar Guidance for Monocular Depth Estimationcited by 379
  6. 2022Generalized Decoding for Pixel, Image, and Languagecited by 292
  7. 2019Attention on Attention for Image Captioningcited by 289
  8. 2022GLIPv2: Unifying Localization and Vision-Language Understandingcited by 275
  9. 2014Synthetic Data and Artificial Neural Networks for Natural Scene Text Recognitioncited by 214
  10. 2022Focal Modulation Networkscited by 186
  11. 2022BinsFormer: Revisiting Adaptive Bins for Monocular Depth Estimationcited by 166
  12. 2020SEED: Semantics Enhanced Encoder-Decoder Framework for Scene Text Recognitioncited by 119
  13. 2020Captioning Images Taken by People Who Are Blindcited by 108
  14. 2022Language Models are General-Purpose Interfacescited by 96
  15. 2022Global-Local Path Networks for Monocular Depth Estimation with Vertical CutDepthcited by 58
  16. 2022Uni-Perceiver v2: A Generalist Model for Large-Scale Vision and Vision-Language Taskscited by 57
  17. 2022MaskOCR: Text Recognition with Masked Encoder-Decoder Pretrainingcited by 34
  18. 2017Show, Ask, Attend, and Answer: A Strong Baseline For Visual Question Answeringcited by 31
  19. 2021Trans4Trans: Efficient Transformer for Transparent Object Segmentation to Help Visually Impaired People Navigate in the Real Worldcited by 30
  20. 2024Open Panoramic Segmentationcited by 17
  21. 2021GEM: A General Evaluation Benchmark for Multimodal Taskscited by 16
  22. 2023Open Scene Understanding: Grounded Situation Recognition Meets Segment Anything for Helping People with Visual Impairmentscited by 13
  23. 2023MateRobot: Material Recognition in Wearable Robotics for People with Visual Impairmentscited by 9
  24. 2023Salient Object Detection for Images Taken by People With Vision Impairmentscited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 1.

2 papers.

  1. 2025CHAOS: Chart Analysis with Outlier Samplescited by 1
  2. 2026Visual Input and Its Framing Affect Attribute-based Descriptions Produced by Large Vision-Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026VIABench: A Comprehensive Video Benchmark Collected from Blind Individuals for Visual Impairment Assistance
  2. 2023VisIT-Bench: A Benchmark for Vision-Language Instruction Following Inspired by Real-World Use
  3. 2025Bridging vision language model (VLM) evaluation gaps with a framework for scalable and cost-effective benchmark generation
  4. 2024Vision-Language Models under Cultural and Inclusive Considerations
  5. 2025Are Large Vision-Language Models Ready to Guide Blind and Low-Vision Individuals?
  6. 2024HumanVLM: Foundation for Human-Scene Vision-Language Model
  7. 2026Focus Where It Counts: A Salience-Driven Vision-Language Model for Low Vision Assistance
  8. 2024Heron-Bench: A Benchmark for Evaluating Vision Language Models in Japanese