# VIABLE: A Visually Impaired Assistance Benchmark for VLM-as-a-Judge Evaluation

arXiv: https://arxiv.org/abs/2605.31351 · HTML: https://mlsanity.com/paper/2605.31351/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 37 of its 47 references that are in the map.
- First posted on arXiv 29 May 2026.

## Builds on (most cited first, 30 of 37)

- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection](https://mlsanity.com/paper/2303.05499/) (2023)
- [CIDEr: Consensus-based Image Description Evaluation](https://mlsanity.com/paper/1411.5726/) (2014)
- [InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency](https://mlsanity.com/paper/2508.18265/) (2025)
- [A Survey on LLM-as-a-Judge](https://mlsanity.com/paper/2411.15594/) (2024)
- [Depth Anything 3: Recovering the Visual Space from Any Views](https://mlsanity.com/paper/2511.10647/) (2025)
- [LLM Evaluators Recognize and Favor Their Own Generations](https://mlsanity.com/paper/2404.13076/) (2024)
- [RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback](https://mlsanity.com/paper/2309.00267/) (2023)
- [RewardBench: Evaluating Reward Models for Language Modeling](https://mlsanity.com/paper/2403.13787/) (2024)
- [MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmark](https://mlsanity.com/paper/2402.04788/) (2024)
- [Justice or Prejudice? Quantifying Biases in LLM-as-a-Judge](https://mlsanity.com/paper/2410.02736/) (2024)
- [JudgeLM: Fine-tuned Large Language Models are Scalable Judges](https://mlsanity.com/paper/2310.17631/) (2023)
- [JudgeBench: A Benchmark for Evaluating LLM-based Judges](https://mlsanity.com/paper/2410.12784/) (2024)
- [EgoDex: Learning Dexterous Manipulation from Large-Scale Egocentric Video](https://mlsanity.com/paper/2505.11709/) (2025)
- [MiniCPM-V 4.5: Cooking Efficient MLLMs via Architecture, Data, and Training Recipe](https://mlsanity.com/paper/2509.18154/) (2025)
- [Devils in Middle Layers of Large Vision-Language Models: Interpreting, Detecting and Mitigating Object Hallucinations via Attention Lens](https://mlsanity.com/paper/2411.16724/) (2024)
- [Prometheus-Vision: Vision-Language Model as a Judge for Fine-Grained Evaluation](https://mlsanity.com/paper/2401.06591/) (2024)
- [VL-RewardBench: A Challenging Benchmark for Vision-Language Generative Reward Models](https://mlsanity.com/paper/2411.17451/) (2024)
- [Can You Trust LLM Judgments? Reliability of LLM-as-a-Judge](https://mlsanity.com/paper/2412.12509/) (2024)
- [Perspective-Aware Reasoning in Vision-Language Models via Mental Imagery Simulation](https://mlsanity.com/paper/2504.17207/) (2025)
- [LLMs Cannot Reliably Judge (Yet?): A Comprehensive Assessment on the Robustness of LLM-as-a-Judge](https://mlsanity.com/paper/2506.09443/) (2025)
- [Judge Anything: MLLM as a Judge Across Any Modality](https://mlsanity.com/paper/2503.17489/) (2025)
- [PCA-Bench: Evaluating Multimodal Large Language Models in Perception-Cognition-Action Chain](https://mlsanity.com/paper/2402.15527/) (2024)
- [Modeling Fine-Grained Hand-Object Dynamics for Egocentric Video Representation Learning](https://mlsanity.com/paper/2503.00986/) (2025)
- [ViLBench: A Suite for Vision-Language Process Reward Modeling](https://mlsanity.com/paper/2503.20271/) (2025)
- [Youtu-VL: Unleashing Visual Potential via Unified Vision-Language Supervision](https://mlsanity.com/paper/2601.19798/) (2026)
- [Teaching Vision-Language Models to Ask: Resolving Ambiguity in Visual Questions](https://mlsanity.com/paper/2507.13773/) (2025)
- [Guide-LLM: An Embodied LLM Agent and Text-Based Topological Map for Robotic Guidance of People with Visual Impairments](https://mlsanity.com/paper/2410.20666/) (2024)
- [GuideDog: A Real-World Egocentric Multimodal Dataset for Blind and Low-Vision Accessibility-Aware Guidance](https://mlsanity.com/paper/2503.12844/) (2025)
- [Judging the Judges: Can Large Vision-Language Models Fairly Evaluate Chart Comprehension and Reasoning?](https://mlsanity.com/paper/2505.08468/) (2025)

## Built on by (most cited first, 1 of 1)

- [A Low Grounding Score Is Not an Ungrounded Judge: Identifying the Perceptibility Confound in Multimodal Oversight](https://mlsanity.com/paper/2610.00111/) (2026)

## Abstract (arXiv)

AI-based Visually Impaired Assistance (VIA) remains challenging, largely due to the high cost of human evaluation. The VLM-as-a-Judge paradigm may offer a promising alternative, although it has mostly been studied in general domains. We therefore ask whether such judges can be trusted for VIA tasks. To investigate this question, we introduce VIABLE (Visually Impaired Assistance Benchmark for VLM-as-a-Judge Evaluation), the first benchmark for VLM-as-a-Judge evaluation in VIA. VIABLE contains over 300K judgment samples across three scenarios and introduces an Effectiveness--Impartiality--Stability framework with a 12-mode failure taxonomy. Based on VIABLE, our systematic study of seven judges across different model scales shows that existing models are largely unreliable across all evaluation axes. The strongest judge, GPT-5.4, achieves only 52.6% single-failure diagnostic accuracy, yet exhibits the highest self-preference rate at 94.2%; while open-source judges are strongly biased and adversarially fragile. To address these issues, we propose VIA-Judge-Agent, a model-agnostic inference-time harness that augments judges with visual evidence extraction and a taxonomy-guided workflow. It enables positive improvements in diagnostic accuracy and downstream VIA responses more preferred by BLV users. Data and code are available at: https://github.com/YiyiyiZhao/VIABLE