mlsanityData to 7 Oct 2026

MM-CRITIC: A Holistic Evaluation of Large Multimodal Models as Multimodal Critique

arXiv 2511.09067 ↗First posted 12 Nov 2025cs.CL cs.AIcited by 2

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

The ability of critique is vital for models to self-improve and serve as reliable AI assistants. While extensively studied in language-only settings, multimodal critique of Large Multimodal Models (LMMs) remains underexplored despite their growing capabilities in tasks like captioning and visual reasoning. In this work, we introduce MM-CRITIC, a holistic benchmark for evaluating the critique ability of LMMs across multiple dimensions: basic, correction, and comparison. Covering 8 main task types and over 500 tasks, MM-CRITIC collects responses from various LMMs with different model sizes and is composed of 4471 samples. To enhance the evaluation reliability, we integrate expert-informed ground answers into scoring rubrics that guide GPT-4o in annotating responses and generating reference critiques, which serve as anchors for trustworthy judgments. Extensive experiments validate the effectiveness of MM-CRITIC and provide a comprehensive assessment of leading LMMs' critique capabilities under multiple dimensions. Further analysis reveals some key insights, including the correlation between response quality and critique, and varying critique difficulty across evaluation dimensions. Our code is available at https://github.com/MichealZeng0420/MM-Critic.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 29 of its 39 references that are in the map. First posted on arXiv 12 Nov 2025.

Builds on

29 of 39 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025AdamMeme: Adaptively Probe the Reasoning Capacity of Multimodal Large Language Models on Harmfulness2 Jul
  2. 2024Multi-modal Preference Alignment Remedies Degradation of Visual Instruction Tuning on Language Models16 Feb
  3. 2024MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmark7 Feb

Other references · 26

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  2. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  3. 2024VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMscited by 749
  4. 2024JudgeBench: A Benchmark for Evaluating LLM-based Judgescited by 294
  5. 2025ScreenSpot-Pro: GUI Grounding for Professional High-Resolution Computer Usecited by 246
  6. 2023Automatically Correcting Large Language Models: Surveying the landscape of diverse self-correction strategiescited by 236
  7. 2024Large Language Models for Data Annotation and Synthesis: A Surveycited by 201
  8. 2025Multimodal Chain-of-Thought Reasoning: A Comprehensive Surveycited by 176
  9. 2024AuroraCap: Efficient, Performant Video Detailed Captioning and a New Benchmarkcited by 137
  10. 2024Aria-UI: Visual Grounding for GUI Instructionscited by 136
  11. 2025GTA1: GUI Test-time Scaling Agentcited by 102
  12. 2023Shepherd: A Critic for Language Model Generationcited by 96
14 more
  1. 2025MCP-Universe: Benchmarking Large Language Models with Real-World Model Context Protocol Serverscited by 89
  2. 2024CriticBench: Benchmarking LLMs for Critique-Correct Reasoningcited by 88
  3. 2024VL-RewardBench: A Challenging Benchmark for Vision-Language Generative Reward Modelscited by 85
  4. 2024GOAT-Bench: Safety Insights to Large Multimodal Models through Meme-Based Social Abusecited by 50
  5. 2024Critic-V: VLM Critics Help Catch VLM Errors in Multimodal Reasoningcited by 50
  6. 2024MMCode: Benchmarking Multimodal Large Language Models for Code Generation with Visually Rich Programming Problemscited by 38
  7. 2024Diving into Self-Evolving Training for Multimodal Reasoningcited by 31
  8. 2024Towards Low-Resource Harmful Meme Detection with LMM Agentscited by 23
  9. 2025SpaCE-10: A Comprehensive Benchmark for Multimodal Large Language Models in Compositional Spatial Intelligencecited by 22
  10. 2025FACT-AUDIT: An Adaptive Multi-Agent Framework for Dynamic Fact-Checking Evaluation of Large Language Modelscited by 19
  11. 2024VideoAutoArena: An Automated Arena for Evaluating Large Multimodal Models in Video Analysis through User Simulationcited by 18
  12. 2023A Large Language Model Approach to Educational Survey Feedback Analysiscited by 14
  13. 2023On the Intersection of Self-Correction and Trust in Language Modelscited by 12
  14. 2024ScratchEval: Are GPT-4o Smarter than My Child? Evaluating Large Multimodal Models with Visual Programming Challengescited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 1.

2 papers.

  1. 2025Multi-Crit: Benchmarking Multimodal Judges on Pluralistic Criteria-Followingcited by 14
  2. 2026MM-JudgeBias: A Benchmark for Evaluating Compositional Biases in MLLM-as-a-Judgecited by 3

Similar, not cited

Close in wording, no citation either way

  1. 2024LLaVA-Critic: Learning to Evaluate Multimodal Models
  2. 2026MMPCBench: Benchmarking Multimodal Large Language Models on Proactive Critique of Flawed Inputs
  3. 2024MM-InstructEval: Zero-Shot Evaluation of (Multimodal) Large Language Models on Multimodal Reasoning Tasks
  4. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities
  5. 2026Judge Model for Large-scale Multimodality Benchmarks
  6. 2023Beyond Task Performance: Evaluating and Reducing the Flaws of Large Multimodal Models with In-Context Learning
  7. 2026What We are Missing in Multimodal LLM Evaluation?
  8. 2025Can Large Multimodal Models Actively Recognize Faulty Inputs? A Systematic Evaluation Framework of Their Input Scrutiny Ability