mlsanityData to 14 Oct 2026

MLLM-as-a-Judge Exhibits Model Preference Bias

arXiv 2604.11589 ↗First posted 13 Apr 2026cs.CVcited by 1

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Automatic evaluation using multimodal large language models (MLLMs), commonly referred to as MLLM-as-a-Judge, has been widely used to measure model performance. If such MLLM-as-a-Judge methods were biased, they could distort model comparisons and benchmark-driven scientific progress. However, it remains unclear to what extent MLLM-as-a-Judge methods favor or disfavor text generated by specific MLLMs. In this study, we propose Philautia-Eval to investigate such model-specific preference bias. Philautia-Eval quantifies the degree of the bias by disentangling preference tendencies from differences in generation quality. Using 1.29M caption-score pairs collected from 12 MLLMs, we found that representative MLLMs tend to exhibit self-preference bias. Moreover, experimental results indicate mutual preference bias within particular model families, which is potentially driven by reused connectors and overlapping instruction-tuning resources. Finally, we introduce a simple ensemble of MLLMs, Pomms. Our results demonstrated that Pomms effectively mitigated the model-specific preference bias while maintaining performance.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 53 of its 61 references that are in the map. First posted on arXiv 13 Apr 2026.

Builds on

53 of 61 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024DeepSeek-VL2: Mixture-of-Experts Vision-Language Models for Advanced Multimodal Understanding13 Dec
  2. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling6 Dec
  3. 2024LLaVA-OneVision: Easy Visual Task Transfer6 Aug
  4. 2024Efficient Multimodal Large Language Models: A Survey17 May
  5. 2024MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmark7 Feb
  6. 2023Improved Baselines with Visual Instruction Tuning5 Oct
  7. 2023A Survey on Multimodal Large Language Models23 Jun

Other references · 46

  1. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  3. 2025Qwen2.5-VL Technical Reportcited by 5,587
  4. 2024GPT-4o System Cardcited by 4,588
  5. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,796
  6. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
  7. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,917
  8. 2025Gemma 3 Technical Reportcited by 1,824
  9. 2024A Survey on LLM-as-a-Judgecited by 1,423
  10. 2023Large Language Models are not Fair Evaluatorscited by 1,085
  11. 2020CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Modelscited by 710
  12. 2024LLM Evaluators Recognize and Favor Their Own Generationscited by 686
34 more
  1. 2023Bias and Fairness in Large Language Models: A Surveycited by 639
  2. 2024LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methodscited by 465
  3. 2018nocaps: novel object captioning at scalecited by 444
  4. 2023JudgeLM: Fine-tuned Large Language Models are Scalable Judgescited by 367
  5. 2023GPTScore: Evaluate as You Desirecited by 361
  6. 2024Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Modelscited by 329
  7. 2024Humans or LLMs as the Judge? A Study on Judgement Biasescited by 323
  8. 2024Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judgecited by 269
  9. 2024Self-Preference Bias in LLM-as-a-Judgecited by 258
  10. 2024Towards Measuring and Modeling "Culture" in LLMs: A Surveycited by 167
  11. 2024Pride and Prejudice: LLM Amplifies Self-Bias in Self-Refinementcited by 114
  12. 2024CheckEval: A reliable LLM-as-a-Judge framework for evaluating text generation using checklistscited by 94
  13. 2023LLMs as Narcissistic Evaluators: When Ego Inflates Evaluation Scorescited by 90
  14. 2023Positive-Augmented Contrastive Learning for Image and Video Captioning Evaluationcited by 78
  15. 2025Eagle 2: Building Post-Training Data Strategies from Scratch for Frontier Vision-Language Modelscited by 70
  16. 2024Understanding the Capabilities and Limitations of Large Language Models for Cultural Commonsensecited by 51
  17. 2024Explaining Length Bias in LLM-Based Preference Evaluationscited by 49
  18. 2024Polos: Multimodal Metric Learning from Human Feedback for Image Captioningcited by 46
  19. 2024FLEUR: An Explainable Reference-Free Evaluation Metric for Image Captioning Using a Large Multimodal Modelcited by 45
  20. 2023CLAIR: Evaluating Image Captions with Large Language Modelscited by 43
  21. 2025Do LLM Evaluators Prefer Themselves for a Reason?cited by 38
  22. 2015From Images to Sentences through Scene Description Graphs using Commonsense Reasoning and Knowledgecited by 35
  23. 2025Beyond the Surface: Measuring Self-Preference in LLM Judgmentscited by 31
  24. 2024A Comprehensive Survey of Bias in LLMs: Current Landscape and Future Directionscited by 27
  25. 2024AI-AI Bias: large language models favor communications generated by large language modelscited by 26
  26. 2025From Perception to Cognition: A Survey of Vision-Language Interactive Reasoning in Multimodal Large Language Modelscited by 26
  27. 2024G-VEval: A Versatile Metric for Evaluating Image and Video Captions Using GPT-4ocited by 23
  28. 2024Positive-Augmented Contrastive Learning for Vision-and-Language Evaluation and Trainingcited by 16
  29. 2024DENEB: A Hallucination-Robust Automatic Evaluation Metric for Image Captioningcited by 11
  30. 2024Likelihood-based Mitigation of Evaluation Bias in Large Language Modelscited by 9
  31. 2025EXPERT: An Explainable Image Captioning Evaluation Metric with Structured Explanationscited by 7
  32. 2025VELA: An LLM-Hybrid-as-a-Judge Approach for Evaluating Long Image Captionscited by 5
  33. 2025DISCODE: Distribution-Aware Score Decoder for Robust Automatic Evaluation of Image Captioningcited by 2
  34. 2025LLM-Free Image Captioning Evaluation in Reference-Flexible Settingscited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Do MLLM Judges Judge the Edit? Auditing Bias in Image Editing Evaluation with Verified Quality Preservationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026MM-JudgeBias: A Benchmark for Evaluating Compositional Biases in MLLM-as-a-Judge
  2. 2023MLLM-Bench: Evaluating Multimodal LLMs with Per-sample Criteria
  3. 2025Aligning MLLM Benchmark With Human Preferences via Structural Equation Modeling
  4. 2026JudgeArena: A Unified Framework for Reproducible LLM-Judge Evaluation
  5. 2024Fairer Preferences Elicit Improved Human-Aligned Large Language Model Judgments
  6. 2025Breaking the Mirror: Activation-Based Mitigation of Self-Preference in LLM Evaluators
  7. 2026Advancing Multimodal Judge Models through a Capability-Oriented Benchmark and MCTS-Driven Data Generation
  8. 2024Direct Judgement Preference Optimization