When the Judge Acts: Auditing VLM-Guided Image Selection on Culturally Situated Prompts
On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning also fits: Social bias and fairness in VLMs
Abstract (arXiv)
Vision-language models (VLMs) increasingly act as judges that pick the best of several generated images, so their choices decide what users see. Such judges are usually validated by score agreement with human ratings, not by the images they return. We audit VLM judges as decision-makers: on 300 culturally situated prompts, we compare the returned image with human ratings the judge never sees and with random choice from the same candidates, and repeat every decision with the candidates reordered. A 4B-parameter judge barely beats random and falls short of a CLIP similarity baseline. It picks the first image shown in 49% of calls (chance: 28%), and reordering changes its choice on 60% of prompts. For this judge, agreement across orders is informative: decisions that survive reordering are much better than random, whereas agreement with a weaker second judge keeps the wrong ones. An 8B judge shows almost no position bias and outperforms CLIP, yet for it the same filter mostly discards good decisions. Agreement helps only when it targets the judge's failure mode, so filters must be re-audited whenever the judge changes. The 4B judge's slight rise in stereotype ratings is no longer detectable after aggregating across orders or with the larger judge.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 14 of its 19 references that are in the map. First posted on arXiv 1 Oct 2026.
Builds on
14 of 19 references.
Closest ancestors · 2
Same direction, newest first
- 2025Fooling the LVLM Judges: Visual Biases in LVLM-Based Evaluation21 May
- 2024Selective "Selective Prediction": Reducing Unnecessary Abstention in Vision-Language Reasoning23 Feb
Other references · 12
- 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
- 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,388
- 2023Large Language Models are not Fair Evaluatorscited by 1,050
- 2017Selective Classification for Deep Neural Networkscited by 799
- 2024MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmarkcited by 362
- 2026JudgeSense: A Benchmark for Prompt Sensitivity in LLM-as-a-Judge Systemscited by 10
- 2026BAS: A Decision-Theoretic Approach to Evaluating Large Language Model Confidencecited by 5
- 2026K-Sort Eval: Efficient Preference Evaluation for Visual Generation via Corrected VLM-as-a-Judgecited by 3
- 2026Multimodal Reward Hacking in Reinforcement Learningcited by 2
- 2026Toward Trustworthy Portrait Editing: Evaluation of Demographic Misrepresentation in I2I Modelscited by 1
- 2026Debiasing Text-to-Image Evaluation via Implicit Cultural Alignment Reward Modelingcited by 1
- 2026Prompt Revision as a Source of Cultural Bias in Text-to-Image Systemscited by 1
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026When Vision-Language Models Judge Without Seeing: Exposing Informativeness Bias
- 2026VLM Judges Can Rank but Cannot Score: Task-Dependent Uncertainty in Multimodal Evaluation
- 2025Is Your Video Language Model a Reliable Judge?
- 2026Feed the Panel Dimensions, Not Verdicts: Rubric-Decomposed Fusion of Vision-Language Aesthetic Judges
- 2026Do MLLM Judges Judge the Edit? Auditing Bias in Image Editing Evaluation with Verified Quality Preservation
- 2026Lost in Translation: Do LVLM Judges Generalize Across Languages?
- 2025Self-Improving VLM Judges Without Human Annotations
- 2025Judging by Appearances? Auditing and Intervening Vision-Language Models for Bail Prediction