mlsanityData to 14 Oct 2026

Calibrating MLLM-as-a-judge via Multimodal Bayesian Prompt Ensembles

arXiv 2509.08777 ↗First posted 10 Sep 2025cs.CV cs.CLcited by 2

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Multimodal large language models (MLLMs) are increasingly used to evaluate text-to-image (TTI) generation systems, providing automated judgments based on visual and textual context. However, these "judge" models often suffer from biases, overconfidence, and inconsistent performance across diverse image domains. While prompt ensembling has shown promise for mitigating these issues in unimodal, text-only settings, our experiments reveal that standard ensembling methods fail to generalize effectively for TTI tasks. To address these limitations, we propose a new multimodal-aware method called Multimodal Mixture-of-Bayesian Prompt Ensembles (MMB). Our method uses a Bayesian prompt ensemble approach augmented by image clustering, allowing the judge to dynamically assign prompt weights based on the visual characteristics of each sample. We show that MMB improves accuracy in pairwise preference judgments and greatly enhances calibration, making it easier to gauge the judge's true uncertainty. In evaluations on two TTI benchmarks, HPSv2 and MJBench, MMB outperforms existing baselines in alignment with human annotations and calibration across varied image content. Our findings highlight the importance of multimodal-specific strategies for judge calibration and suggest a promising path forward for reliable large-scale TTI evaluation.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 52 of its 75 references that are in the map. First posted on arXiv 10 Sep 2025.

Builds on

52 of 75 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024LLaVA-Critic: Learning to Evaluate Multimodal Models3 Oct
  2. 2023VisIT-Bench: A Benchmark for Vision-Language Instruction Following Inspired by Real-World Use12 Aug
  3. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning11 May

Other references · 49

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  5. 2023Visual Instruction Tuningcited by 7,869
  6. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  7. 2017On Calibration of Modern Neural Networkscited by 4,960
  8. 2024GPT-4o System Cardcited by 4,588
  9. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,356
  10. 2016Improved Techniques for Training GANscited by 2,970
  11. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  12. 2015VQA: Visual Question Answeringcited by 2,506
37 more
  1. 2017Billion-scale similarity search with GPUscited by 2,497
  2. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,224
  3. 2024A Survey on LLM-as-a-Judgecited by 1,423
  4. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,089
  5. 2023Large Language Models are not Fair Evaluatorscited by 1,085
  6. 2023Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generationcited by 977
  7. 2019How Can We Know What Language Models Know?cited by 959
  8. 2019Improved Precision and Recall Metric for Assessing Generative Modelscited by 944
  9. 2023Human Preference Score v2: A Solid Benchmark for Evaluating Human Preferences of Text-to-Image Synthesiscited by 890
  10. 2017Selective Classification for Deep Neural Networkscited by 852
  11. 2016SPICE: Semantic Propositional Image Caption Evaluationcited by 824
  12. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 781
  13. 2024LLM Evaluators Recognize and Favor Their Own Generationscited by 686
  14. 2023A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPTcited by 619
  15. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 565
  16. 2024From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judgecited by 494
  17. 2024LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methodscited by 465
  18. 2023Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuningcited by 445
  19. 2016Visual Dialogcited by 377
  20. 2024Justice or Prejudice? Quantifying Biases in LLM-as-a-Judgecited by 368
  21. 2017Predict Responsibly: Improving Fairness and Accuracy by Learning to Defercited by 233
  22. 2022DALL-Eval: Probing the Reasoning Skills and Social Biases of Text-to-Image Generation Modelscited by 230
  23. 2023LAMM: Language-Assisted Multi-Modal Instruction-Tuning Dataset, Framework, and Benchmarkcited by 169
  24. 2024GPT-4V(ision) is a Human-Aligned Evaluator for Text-to-3D Generationcited by 153
  25. 2023Benchmarking Cognitive Biases in Large Language Models as Evaluatorscited by 150
  26. 2023GPT-4V(ision) as a Generalist Evaluator for Vision-Language Taskscited by 119
  27. 2021Machine Learning with a Reject Option: A surveycited by 109
  28. 2023Verbosity Bias in Preference Labeling by Large Language Modelscited by 91
  29. 2023LLMs as Narcissistic Evaluators: When Ego Inflates Evaluation Scorescited by 90
  30. 2022Reliable Visual Question Answering: Abstain Rather Than Answer Incorrectlycited by 89
  31. 2024MJ-Bench: Is Your Multimodal Reward Model Really a Good Judge for Text-to-Image Generation?cited by 66
  32. 2022On the Blind Spots of Model-Based Evaluation Metrics for Text Generationcited by 49
  33. 2022PromptBoosting: Black-Box Text Classification with Ten Forward Passescited by 36
  34. 2024Are LLM-based Evaluators Confusing NLG Quality Criteria?cited by 36
  35. 2024ViGoR: Improving Visual Grounding of Large Vision Language Models with Fine-Grained Reward Modelingcited by 34
  36. 2023X-IQE: eXplainable Image Quality Evaluation for Text-to-Image Generation with Visual Large Language Modelscited by 28
  37. 2023Improving Selective Visual Question Answering by Learning from Your Peerscited by 25

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026H-Adapter: Pose-Robust Hairstyle Transfer via Attention-Derived, Source-Aligned Hair Maskscited by 0
  2. 2026Uncertainty-Aware Decision Making in Multimodal Large Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Bi-Level Prompt Optimization for Multimodal LLM-as-a-Judge
  2. 2025Multimodal Prompt Optimization: Why Not Leverage Multiple Modalities for MLLMs
  3. 2026MM-JudgeBias: A Benchmark for Evaluating Compositional Biases in MLLM-as-a-Judge
  4. 2025Test-time Prompt Refinement for Text-to-Image Models
  5. 2025Aligning Multimodal LLM with Human Preference: A Survey
  6. 2024MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmark
  7. 2024A Comprehensive Study of Multimodal Large Language Models for Image Quality Assessment
  8. 2024Unveiling Uncertainty: A Deep Dive into Calibration and Performance of Multimodal Large Language Models