mlsanityData to 7 Oct 2026

Self-Improving VLM Judges Without Human Annotations

arXiv 2512.05145 ↗First posted 2 Dec 2025cs.CVcited by 7

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

also in 1 more
Abstract (arXiv)

Effective judges of Vision-Language Models (VLMs) are crucial for model development. Current methods for training VLM judges mainly rely on large-scale human preference annotations. However, such an approach is costly, and the annotations easily become obsolete as models rapidly improve. In this work, we present a framework to self-train a VLM judge model without any human preference annotations, using only self-synthesized data. Our method is iterative and has three stages: (1) generate diverse multimodal instruction-response pairs at varying quality levels, (2) generate reasoning traces and judgments for each pair, removing the ones that do not match our expected quality levels, and (3) training on correct judge answers and their reasoning traces. We evaluate the resulting judge on Multimodal RewardBench and VL-RewardBench across domains: correctness, preference, reasoning, safety, and visual question-answering. Our method improves a Llama-3.2-11B multimodal judge from 0.38 to 0.51 in overall accuracy on VL-RewardBench, often outperforming much larger models including Llama-3.2-90B, GPT-4o, and Claude 3.5 Sonnet, with particularly strong gains in general, hallucination, and reasoning dimensions. The overall strength of these human-annotation-free results suggest the potential for a future self-judge that evolves alongside rapidly improving VLM capabilities.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 32 of its 38 references that are in the map. First posted on arXiv 2 Dec 2025.

Builds on

32 of 38 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025Multimodal RewardBench: Holistic Evaluation of Reward Models for Vision Language Models20 Feb
  2. 2024LLaVA-Critic: Learning to Evaluate Multimodal Models3 Oct
  3. 2024MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmark7 Feb
  4. 2023Aligning Large Multimodal Models with Factually Augmented RLHF25 Sep
  5. 2023VisIT-Bench: A Benchmark for Vision-Language Instruction Following Inspired by Real-World Use12 Aug

Other references · 27

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  3. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
  4. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  5. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  6. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 988
  7. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decodingcited by 671
  8. 2024Self-Rewarding Language Modelscited by 654
  9. 2018nocaps: novel object captioning at scalecited by 444
  10. 2023Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuningcited by 439
  11. 2023UltraFeedback: Boosting Language Models with Scaled AI Feedbackcited by 394
  12. 2023Analyzing and Mitigating Object Hallucination in Large Vision-Language Modelscited by 314
15 more
  1. 2023Self-Consuming Generative Models Go MADcited by 239
  2. 2024Aligning Modalities in Vision Large Language Models via Preference Fine-tuningcited by 197
  3. 2025Unified Reward Model for Multimodal Understanding and Generationcited by 194
  4. 2024RLAIF-V: Open-Source AI Feedback Leads to Super GPT-4V Trustworthinesscited by 129
  5. 2025Preference Leakage: A Contamination Problem in LLM-as-a-judgecited by 119
  6. 2023GPT-4V(ision) as a Generalist Evaluator for Vision-Language Taskscited by 118
  7. 2024Synthetic Data (Almost) from Scratch: Generalized Instruction Tuning for Language Modelscited by 99
  8. 2024VL-RewardBench: A Challenging Benchmark for Vision-Language Generative Reward Modelscited by 85
  9. 2024WildVision: Evaluating Vision-Language Models in the Wild with Human Preferencescited by 76
  10. 2025Learning to Plan & Reason for Evaluation with Thinking-LLM-as-a-Judgecited by 75
  11. 2025J1: Incentivizing Thinking in LLM-as-a-Judge via Reinforcement Learningcited by 73
  12. 2023Visual Program Distillation: Distilling Tools and Programmatic Reasoning into Vision-Language Modelscited by 66
  13. 2024VLFeedback: A Large-Scale AI Feedback Dataset for Large Vision-Language Models Alignmentcited by 56
  14. 2024Self-Improving Diffusion Models with Synthetic Datacited by 34
  15. 2024FGAIF: Aligning Large Vision-Language Models with Fine-grained AI Feedbackcited by 30

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 7.

7 papers.

  1. 2026On the Generalization Gap in Self-Evolving Language Model Reasoningcited by 4
  2. 2026OpenVTON-Bench: A Large-Scale High-Resolution Benchmark for Controllable Virtual Try-On Evaluationcited by 2
  3. 2026Seeing Isn't Believing: Uncovering Blind Spots in Evaluator Vision-Language Modelscited by 1
  4. 2026VIABLE: A Visually Impaired Assistance Benchmark for VLM-as-a-Judge Evaluationcited by 1
  5. 2026VERIA: Verification-Centric Multimodal Instance Augmentation for Long-Tailed 3D Object Detectioncited by 0
  6. 2026VLM-in-the-Loop: A Plug-In Quality Assurance Module for ECG Digitization Pipelinescited by 0
  7. 2026Beyond Binary: Reframing GUI Critique as Continuous Semantic Alignmentcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Lost in Translation: Do LVLM Judges Generalize Across Languages?
  2. 2025VL-GenRM: Enhancing Vision-Language Verification via Vision Experts and Iterative Training
  3. 2026Multi-Task Reinforcement Learning for Enhanced Multimodal LLM-as-a-Judge
  4. 2026V-Zero: Self-Improving Multimodal Reasoning with Zero Annotation
  5. 2025Improve LLM-as-a-Judge Ability as a General Ability
  6. 2024Critic-V: VLM Critics Help Catch VLM Errors in Multimodal Reasoning
  7. 2026When Vision-Language Models Judge Without Seeing: Exposing Informativeness Bias
  8. 2025Iterative Self-Improvement of Vision Language Models for Image Scoring and Self-Explanation