mlsanityData to 7 Oct 2026

Multi-Crit: Benchmarking Multimodal Judges on Pluralistic Criteria-Following

arXiv 2511.21662 ↗First posted 26 Nov 2025cs.CVcited by 14

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Large multimodal models (LMMs) are increasingly adopted as judges in multimodal evaluation systems due to their strong instruction following and consistency with human preferences. However, their ability to follow diverse, fine-grained evaluation criteria remains underexplored. We develop Multi-Crit, a benchmark for evaluating multimodal judges on their capacity to follow pluralistic criteria and produce reliable criterion-level judgments. Covering both open-ended generation and verifiable reasoning tasks, Multi-Crit is built through a rigorous data curation pipeline that gathers challenging response pairs with multi-criterion human annotations. It further introduces three novel metrics for systematically assessing pluralistic adherence, criterion-switching flexibility, and the ability to recognize criterion-level preference conflicts. Comprehensive analysis of 25 LMMs reveals that 1) proprietary models still struggle to maintain consistent adherence to pluralistic criteria--especially in open-ended evaluation; 2) open-source models lag further behind in flexibly following diverse criteria; and 3) critic fine-tuning with holistic judgment signals enhances visual grounding but fails to generalize to pluralistic criterion-level judgment. Additional analyses on reasoning fine-tuning, test-time scaling, and boundary consistency between open-source and proprietary models further probe the limits of current multimodal judges. As a pioneering study, Multi-Crit lays the foundation for building reliable and steerable multimodal AI evaluation.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 62 of its 76 references that are in the map. First posted on arXiv 26 Nov 2025.

Builds on

62 of 76 references.

Closest ancestors · 15

Same direction, newest first

  1. 2025MM-CRITIC: A Holistic Evaluation of Large Multimodal Models as Multimodal Critique12 Nov
  2. 2025LLaVA-OneVision-1.5: Fully Open Framework for Democratized Multimodal Training28 Sep
  3. 2025BaseReward: A Strong Baseline for Multimodal Reward Model19 Sep
  4. 2025MiniCPM-V 4.5: Cooking Efficient MLLMs via Architecture, Data, and Training Recipe16 Sep
  5. 2025Multimodal RewardBench: Holistic Evaluation of Reward Models for Vision Language Models20 Feb
  6. 2025MM-RLHF: The Next Step Forward in Multimodal LLM Alignment14 Feb
  7. 2025InternLM-XComposer2.5-Reward: A Simple Yet Effective Multi-Modal Reward Model21 Jan
  8. 2025Can MLLMs Reason in Multimodality? EMMA: An Enhanced MultiModal ReAsoning Benchmark9 Jan
  9. 2024LLaVA-Critic: Learning to Evaluate Multimodal Models3 Oct
  10. 2024LLaVA-OneVision: Easy Visual Task Transfer6 Aug
  11. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs24 Jun
  12. 2024MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmark7 Feb
3 more
  1. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks21 Dec
  2. 2023Aligning Large Multimodal Models with Factually Augmented RLHF25 Sep
  3. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities4 Aug

Other references · 47

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2025Qwen3 Technical Reportcited by 8,195
  4. 2023Visual Instruction Tuningcited by 7,792
  5. 2025Qwen2.5-VL Technical Reportcited by 5,418
  6. 2024GPT-4o System Cardcited by 4,467
  7. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,655
  8. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,769
  9. 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,419
  10. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,414
  11. 2016Modeling Context in Referring Expressionscited by 1,054
  12. 2024MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?cited by 573
35 more
  1. 2023HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Modelscited by 572
  2. 2024RewardBench: Evaluating Reward Models for Language Modelingcited by 480
  3. 2025GLM-4.5V and GLM-4.1V-Thinking: Towards Versatile Multimodal Reasoning with Scalable Reinforcement Learningcited by 439
  4. 2025Kimi-VL Technical Reportcited by 381
  5. 2024Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Modelscited by 321
  6. 2024JudgeBench: A Benchmark for Evaluating LLM-based Judgescited by 294
  7. 2025Improving Video Generation with Human Feedbackcited by 267
  8. 2025SFT or RL? An Early Investigation into Training R1-Like Reasoning Large Vision-Language Modelscited by 209
  9. 2025Unified Reward Model for Multimodal Understanding and Generationcited by 194
  10. 2025SoTA with Less: MCTS-Guided Sample Selection for Data-Efficient Visual Reasoning Self-Improvementcited by 144
  11. 2024DOCCI: Descriptions of Connected and Contrasting Imagescited by 130
  12. 2024RLAIF-V: Open-Source AI Feedback Leads to Super GPT-4V Trustworthinesscited by 129
  13. 2024Prometheus-Vision: Vision-Language Model as a Judge for Fine-Grained Evaluationcited by 123
  14. 2025MME-CoT: Benchmarking Chain-of-Thought in Large Multimodal Models for Reasoning Quality, Robustness, and Efficiencycited by 123
  15. 2025Are VLMs Ready for Autonomous Driving? An Empirical Study from the Reliability, Data, and Metric Perspectivescited by 120
  16. 2023GPT-4V(ision) as a Generalist Evaluator for Vision-Language Taskscited by 118
  17. 2025Unified Multimodal Chain-of-Thought Reward Model through Reinforcement Fine-Tuningcited by 86
  18. 2024VL-RewardBench: A Challenging Benchmark for Vision-Language Generative Reward Modelscited by 85
  19. 2024WildVision: Evaluating Vision-Language Models in the Wild with Human Preferencescited by 76
  20. 2024Enhancing Visual-Language Modality Alignment in Large Vision Language Models via Self-Improvementcited by 74
  21. 2025R1-Reward: Training Multimodal Reward Model Through Stable Reinforcement Learningcited by 65
  22. 2025Eagle 2.5: Boosting Long-Context Post-Training for Frontier Vision-Language Modelscited by 62
  23. 2024RMB: Comprehensively Benchmarking Reward Models in LLM Alignmentcited by 60
  24. 2025Parallel-R1: Towards Parallel Thinking via Reinforcement Learningcited by 59
  25. 2024ImageInWords: Unlocking Hyper-Detailed Image Descriptionscited by 56
  26. 2024VLFeedback: A Large-Scale AI Feedback Dataset for Large Vision-Language Models Alignmentcited by 56
  27. 2025VisualPuzzles: Decoupling Multimodal Reasoning Evaluation from Domain Knowledgecited by 52
  28. 2025LLaVA-Critic-R1: Your Critic Model is Secretly a Strong Policy Modelcited by 34
  29. 2025Painting with Words: Elevating Detailed Image Captioning with Benchmark and Alignment Learningcited by 23
  30. 2024Improving Context-Aware Preference Modeling for Language Modelscited by 22
  31. 2025Learning to Reason via Mixture-of-Thought for Logical Reasoningcited by 20
  32. 2025ARGUS: Hallucination and Omission Evaluation in Video-LLMscited by 16
  33. 2025Cost-Aware Contrastive Routing for LLMscited by 16
  34. 2023GPT-4 Vision on Medical Image Classification -- A Case Study on COVID-19 Datasetcited by 7
  35. 2025RLBFF: Binary Flexible Feedback to bridge between Human Feedback & Verifiable Rewardscited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 13.

14 papers.

Sort:

By area:

  1. 2026Claw-Eval: Towards Trustworthy Evaluation of Autonomous Agentscited by 57
  2. 2026Parallel-Probe: Towards Efficient Parallel Thinking via 2D Probingcited by 13
  3. 2026Unified Personalized Reward Model for Vision Generationcited by 11
  4. 2026MUSE: Benchmarking Manufacturable, Functional, and Assemblable Text-to-CAD Generationcited by 8
  5. 2025GroundingME: Exposing the Visual Grounding Gap in MLLMs through Multi-Dimensional Evaluationcited by 7
  6. 2026Orchard: An Open-Source Agentic Modeling Frameworkcited by 7
  7. 2026Visual Preference Optimization with Rubric Rewardscited by 5
  8. 2026MM-JudgeBias: A Benchmark for Evaluating Compositional Biases in MLLM-as-a-Judgecited by 3
  9. 2026Unsafe by Reciprocity: How Generation-Understanding Coupling Undermines Safety in Unified Multimodal Modelscited by 2
  10. 2026Reinforcement Learning with Robust Rubric Rewardscited by 2
  11. 2026Mitigating Factual Hallucination in Large Reasoning Models via Mixed-Mode Advantage Regularizationcited by 2
  12. 2026Learning from Self-Debate: Preparing Reasoning Models for Multi-Agent Debatecited by 1
2 more
  1. 2026EviSafe: Evidence-Grounded Safety Evaluation for Vision-Language Modelscited by 1
  2. 2026Video Understanding Reward Modeling: A Robust Benchmark and Performant Reward Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Judge Model for Large-scale Multimodality Benchmarks
  2. 2025Flex-Judge: Text-Only Reasoning Unleashes Zero-Shot Multimodal Evaluators
  3. 2026MJ1: Multimodal Judgment via Grounded Verification
  4. 2026VLM Judges Can Rank but Cannot Score: Task-Dependent Uncertainty in Multimodal Evaluation
  5. 2026Advancing Multimodal Judge Models through a Capability-Oriented Benchmark and MCTS-Driven Data Generation
  6. 2026DeltaRubric: Generative Multimodal Reward Modeling via Joint Planning and Verification
  7. 2025MM-IQ: Benchmarking Human-Like Abstraction and Reasoning in Multimodal Models
  8. 2026Mitigating Perceptual Judgment Bias in Multimodal LLM-as-a-Judge via Perceptual Perturbation and Reward Modeling