mlsanityData to 7 Oct 2026

Feed the Panel Dimensions, Not Verdicts: Rubric-Decomposed Fusion of Vision-Language Aesthetic Judges

arXiv 2609.27110 ↗First posted 22 Sep 2026cs.CV cs.CL cs.LGcited by 0

On the map: Multimodal LLMs / vision-language models → Social bias and fairness in VLMs also fits: VLM benchmarks and visual reasoning

Abstract (arXiv)

Vision-language models (VLMs) are deployed as zero-shot judges of image aesthetics, and panels of several models are recommended, on thin evidence, as the way to make such judges reliable. On two human-rated datasets, EVA and PARA, we find that a panel of holistic judges never significantly beats its best member, whether the verdicts are averaged or fused by a learned combiner. What a panel is worth depends on what it is fed. We therefore have each model score each image on the five dimensions of a frozen, human-written rubric and fuse those scores, alongside each model's verdict, across model families with an out-of-fold combiner. The dimension scores measure what their labels claim: with the overall human score partialled out, a dimension prompt carries more attribute-specific information than the holistic prompt in 28 of 30 model-attribute cells. Fused, they beat the best single VLM in all ten three-family panels on EVA (against that best single model, +0.07 Spearman rho for the strongest trio and +0.10 for the pre-declared one, and +0.06 and +0.07 when averaged over twenty fold partitions; against the panel mean, the primary test gives +0.118 on its EVA design set), and on PARA they reach parity under Spearman rho and a small, non-significant loss under Kendall tau-b, where one model already captures 85% of the human noise ceiling. It is not a feature-count artefact: giving the same combiner an equal number of pure holistic columns, split from the same repetitions, does not reproduce it. The gain costs a few hundred labels, which do not transfer between datasets, and 4.8x the API calls on EVA; we report it with paired bootstraps and Kendall tau-b, alongside a failed pre-registration and the configurations that lost.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Social bias and fairness in VLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 27 of its 41 references that are in the map. First posted on arXiv 22 Sep 2026.

Builds on

27 of 41 references.

References · 27

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  2. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
  3. 2021MUSIQ: Multi-scale Image Quality Transformercited by 1,058
  4. 2022Exploring CLIP for Assessing the Look and Feel of Imagescited by 974
  5. 2023Q-Align: Teaching LMMs for Visual Scoring via Discrete Text-Defined Levelscited by 489
  6. 2024Mixture-of-Agents Enhances Large Language Model Capabilitiescited by 420
  7. 2017NIMA: Neural Image Assessmentcited by 387
  8. 2024MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmarkcited by 362
  9. 2024Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Modelscited by 301
  10. 2023Q-Bench: A Benchmark for General-Purpose Foundation Models on Low-level Visioncited by 247
  11. 2023FLASK: Fine-grained Language Model Evaluation based on Alignment Skill Setscited by 198
  12. 2023Bias Runs Deep: Implicit Reasoning Biases in Persona-Assigned LLMscited by 191
15 more
  1. 2024Prometheus-Vision: Vision-Language Model as a Judge for Fine-Grained Evaluationcited by 123
  2. 2016Photo Aesthetics Ranking Network with Attributes and Content Adaptationcited by 122
  3. 2023Branch-Solve-Merge Improves Large Language Model Evaluation and Generationcited by 104
  4. 2023VILA: Learning Image Aesthetics from User Comments with Vision-Language Pretrainingcited by 63
  5. 2024AesBench: An Expert Benchmark for Multimodal Large Language Models on Image Aesthetics Perceptioncited by 63
  6. 2022Personalized Image Aesthetics Assessment with Rich Attributescited by 56
  7. 2023When "A Helpful Assistant" Is Not Really Helpful: Personas in System Prompts Do Not Improve Performances of Large Language Modelscited by 50
  8. 2024UNIAA: A Unified Multi-modal Image Aesthetic Assessment Baseline and Benchmarkcited by 39
  9. 2026Nine Judges, Two Effective Votes: Correlated Errors Undermine LLM Evaluation Panelscited by 21
  10. 2024DnA-Eval: Enhancing Large Language Model Evaluation through Decomposition and Aggregationcited by 12
  11. 2026Are Diversity Metrics Measuring Diversity? A Capability-Controlled Audit of Majority-Vote Gain in LLM Ensemblescited by 7
  12. 2023Fusion-Eval: Integrating Assistant Evaluators with LLMscited by 6
  13. 2026Can Vision Language Models Assess Graphic Design Aesthetics? A Benchmark, Evaluation, and Dataset Perspectivecited by 6
  14. 2025Who Judges the Judge? LLM Jury-on-Demand: Building Trustworthy LLM Evaluation Systemscited by 5
  15. 2026Multi-Granularity Reasoning for Image Quality Assessment via Attribute-Aware Reinforcement Learning to Rankcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026When Vision-Language Models Judge Without Seeing: Exposing Informativeness Bias
  2. 2026When the Judge Acts: Auditing VLM-Guided Image Selection on Culturally Situated Prompts
  3. 2026VLM Judges Can Rank but Cannot Score: Task-Dependent Uncertainty in Multimodal Evaluation
  4. 2026Visual Aesthetic Benchmark: Can Frontier Models Judge Beauty?
  5. 2026VisionQ: VLM-as-a-Judge Taxonomy, Dataset and Benchmark for Qualitative Analysis in Computer Vision
  6. 2026Opening LLM Judges: Recovering Preference Signals Beyond the Final Verdict
  7. 2025Fooling the LVLM Judges: Visual Biases in LVLM-Based Evaluation
  8. 2026It's Not What the Image Shows: Irrelevant Context Destabilises VLM Judges Without Informing Them