mlsanityData to 14 Oct 2026

VisionQ: VLM-as-a-Judge Taxonomy, Dataset and Benchmark for Qualitative Analysis in Computer Vision

arXiv 2610.00666 ↗First posted 30 Sep 2026cs.CV cs.AIcited by 0

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Qualitative comparison figures are central evidence in computer vision papers, and vision-language models (VLMs) are increasingly used to judge them. Yet existing benchmarks score only scalar quality or overall preference, so a judge can be rewarded for picking the preferred image for the wrong visual reason. We introduce VisionQ, the first benchmark built from peer-reviewed CV comparison figures that grounds every judgment in a named visual criterion: each question states the criterion, and a judge is credited only when it selects the output the authors identify as best on that criterion. We call this task criterion-conditioned visual discrimination. VisionQ comprises (1) a corpus of 1,409 CVPR and ICCV papers with 1,800+ validated comparison figures and 3,911 hand-annotated data points linking method crops to author-stated visual claims; (2) a six-axis, 51-leaf taxonomy of the visual criteria behind qualitative judgment; (3) a criterion-conditioned evaluation protocol that hides method names, captions, and paper identity and reports accuracy per criterion; and (4) VisionQ-Judge, a DPO-tuned Gemma-4-E4B judge trained on symmetric evidence pairs, which reduces last-option predictions by 7.0pp and improves accuracy by 2.5pp on a held-out test set. Evaluating 20 open- and closed-source VLM judges, we find that the strongest reach only 63.1% accuracy (chance 32.2%) and that reliability varies sharply across criteria. Code: https://github.com/ReML-AI/visionq. Data: https://huggingface.co/datasets/visionq-anon-2026/VisionQ-1k.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 27 of its 31 references that are in the map. First posted on arXiv 30 Sep 2026.

Builds on

27 of 31 references.

References · 27

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  3. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,733
  4. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,424
  5. 2023VBench: Comprehensive Benchmark Suite for Video Generative Modelscited by 1,822
  6. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,184
  7. 2023ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generationcited by 1,045
  8. 2022Exploring CLIP for Assessing the Look and Feel of Imagescited by 991
  9. 2023Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generationcited by 977
  10. 2023Human Preference Score v2: A Solid Benchmark for Evaluating Human Preferences of Text-to-Image Synthesiscited by 890
  11. 2020Quantifying Attention Flow in Transformerscited by 775
  12. 2023Prometheus: Inducing Fine-grained Evaluation Capability in Language Modelscited by 575
15 more
  1. 2024BLINK: Multimodal Large Language Models Can See but Not Perceivecited by 550
  2. 2023Q-Align: Teaching LMMs for Visual Scoring via Discrete Text-Defined Levelscited by 503
  3. 2023Large Language Models Are Not Robust Multiple Choice Selectorscited by 486
  4. 2023TIFA: Accurate and Interpretable Text-to-Image Faithfulness Evaluation with Question Answeringcited by 428
  5. 2023Q-Bench: A Benchmark for General-Purpose Foundation Models on Low-level Visioncited by 249
  6. 2023VIEScore: Towards Explainable Metrics for Conditional Image Synthesis Evaluationcited by 201
  7. 2023Davidsonian Scene Graph: Improving Reliability in Fine-grained Evaluation for Text-to-Image Generationcited by 171
  8. 2024LLaVA-Critic: Learning to Evaluate Multimodal Modelscited by 134
  9. 2025VisualQuality-R1: Reasoning-Induced Image Quality Assessment via Reinforcement Learning to Rankcited by 79
  10. 2024Clio: Privacy-Preserving Insights into Real-World AI Usecited by 72
  11. 2024A-Bench: Are LMMs Masters at Evaluating AI-generated Images?cited by 39
  12. 2025Validating LLM-as-a-Judge Systems under Rating Indeterminacycited by 25
  13. 2026Human-Aligned MLLM Judges for Fine-Grained Image Editing Evaluation: A Benchmark, Framework, and Analysiscited by 8
  14. 2026VLM-SubtleBench: How Far Are VLMs from Human-Level Subtle Comparative Reasoning?cited by 4
  15. 2026Advancing Multimodal Judge Models through a Capability-Oriented Benchmark and MCTS-Driven Data Generationcited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025CompareBench: A Benchmark for Visual Comparison Reasoning in Vision-Language Models
  2. 2026VRIQ: Benchmarking and Analyzing Visual-Reasoning IQ of VLMs
  3. 2026When Vision-Language Models Judge Without Seeing: Exposing Informativeness Bias
  4. 2025PairBench: Are Vision-Language Models Reliable at Comparing What They See?
  5. 2025Reasoning in Computer Vision: Taxonomy, Models, Tasks, and Methodologies
  6. 2024How to Determine the Preferred Image Distribution of a Black-Box Vision-Language Model?
  7. 2026UVLM: A Universal Vision-Language Model Loader for Reproducible Multimodal Benchmarking
  8. 2026VB: Visibility Benchmark for Visibility and Perspective Reasoning in Images