VIABLE: A Visually Impaired Assistance Benchmark for VLM-as-a-Judge Evaluation

arXiv 2605.31351 ↗First posted 29 May 2026cs.CL cs.CVcited by 1

Not on the map. Why?

Abstract (arXiv)

AI-based Visually Impaired Assistance (VIA) remains challenging, largely due to the high cost of human evaluation. The VLM-as-a-Judge paradigm may offer a promising alternative, although it has mostly been studied in general domains. We therefore ask whether such judges can be trusted for VIA tasks. To investigate this question, we introduce VIABLE (Visually Impaired Assistance Benchmark for VLM-as-a-Judge Evaluation), the first benchmark for VLM-as-a-Judge evaluation in VIA. VIABLE contains over 300K judgment samples across three scenarios and introduces an Effectiveness--Impartiality--Stability framework with a 12-mode failure taxonomy. Based on VIABLE, our systematic study of seven judges across different model scales shows that existing models are largely unreliable across all evaluation axes. The strongest judge, GPT-5.4, achieves only 52.6% single-failure diagnostic accuracy, yet exhibits the highest self-preference rate at 94.2%; while open-source judges are strongly biased and adversarially fragile. To address these issues, we propose VIA-Judge-Agent, a model-agnostic inference-time harness that augments judges with visual evidence extraction and a taxonomy-guided workflow. It enables positive improvements in diagnostic accuracy and downstream VIA responses more preferred by BLV users. Data and code are available at: https://github.com/YiyiyiZhao/VIABLE

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 37 of its 47 references that are in the map. First posted on arXiv 29 May 2026.

Builds on

37 of 47 references.

References · 37

2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,332
2014CIDEr: Consensus-based Image Description Evaluationcited by 2,046
2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,510
2024A Survey on LLM-as-a-Judgecited by 1,423
2025Depth Anything 3: Recovering the Visual Space from Any Viewscited by 692
2024LLM Evaluators Recognize and Favor Their Own Generationscited by 686
2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackcited by 625
2024RewardBench: Evaluating Reward Models for Language Modelingcited by 499
2024MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmarkcited by 372
2024Justice or Prejudice? Quantifying Biases in LLM-as-a-Judgecited by 368
2023JudgeLM: Fine-tuned Large Language Models are Scalable Judgescited by 367

25 more

2024JudgeBench: A Benchmark for Evaluating LLM-based Judgescited by 318
2025EgoDex: Learning Dexterous Manipulation from Large-Scale Egocentric Videocited by 211
2025MiniCPM-V 4.5: Cooking Efficient MLLMs via Architecture, Data, and Training Recipecited by 186
2024Devils in Middle Layers of Large Vision-Language Models: Interpreting, Detecting and Mitigating Object Hallucinations via Attention Lenscited by 143
2024Prometheus-Vision: Vision-Language Model as a Judge for Fine-Grained Evaluationcited by 125
2024VL-RewardBench: A Challenging Benchmark for Vision-Language Generative Reward Modelscited by 86
2024Can You Trust LLM Judgments? Reliability of LLM-as-a-Judgecited by 56
2025Perspective-Aware Reasoning in Vision-Language Models via Mental Imagery Simulationcited by 42
2025LLMs Cannot Reliably Judge (Yet?): A Comprehensive Assessment on the Robustness of LLM-as-a-Judgecited by 37
2025Judge Anything: MLLM as a Judge Across Any Modalitycited by 34
2024PCA-Bench: Evaluating Multimodal Large Language Models in Perception-Cognition-Action Chaincited by 29
2025Modeling Fine-Grained Hand-Object Dynamics for Egocentric Video Representation Learningcited by 20
2025ViLBench: A Suite for Vision-Language Process Reward Modelingcited by 16
2026Youtu-VL: Unleashing Visual Potential via Unified Vision-Language Supervisioncited by 15
2025Teaching Vision-Language Models to Ask: Resolving Ambiguity in Visual Questionscited by 11
2024Guide-LLM: An Embodied LLM Agent and Text-Based Topological Map for Robotic Guidance of People with Visual Impairmentscited by 10
2025GuideDog: A Real-World Egocentric Multimodal Dataset for Blind and Low-Vision Accessibility-Aware Guidancecited by 10
2025Judging the Judges: Can Large Vision-Language Models Fairly Evaluate Chart Comprehension and Reasoning?cited by 10
2025Praxis-VLM: Vision-Grounded Decision Making via Text-Driven Reinforcement Learningcited by 9
2024Generating Contextually-Relevant Navigation Instructions for Blind and Low Vision Peoplecited by 7
2025Logic Unseen: Revealing the Logical Blindspots of Vision-Language Modelscited by 7
2025Self-Improving VLM Judges Without Human Annotationscited by 7
2025SVLTA: Benchmarking Vision-Language Temporal Alignment via Synthetic Video Situationcited by 4
2025LaF-GRPO: In-Situ Navigation Instruction Generation for the Visually Impaired via GRPO with LLM-as-Follower Rewardcited by 3
2025Less Redundancy: Boosting Practicality of Vision Language Model in Walking Assistantscited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

2026A Low Grounding Score Is Not an Ungrounded Judge: Identifying the Perceptibility Confound in Multimodal Oversightcited by 0

Similar, not cited

Close in wording, no citation either way