mlsanityData to 7 Oct 2026

Learning What Matters: Dynamic Dimension Selection and Aggregation for Interpretable Vision-Language Reward Modeling

arXiv 2604.05445 ↗First posted 7 Apr 2026cs.CL cs.AI cs.CVcited by 0

On the map: LLM alignment / RLHF / preference optimisation → Diffusion and multimodal model alignment

Abstract (arXiv)

Vision-language reward modeling faces a dilemma: generative approaches are interpretable but slow, while discriminative ones are efficient but act as opaque "black boxes." To bridge this gap, we propose VL-MDR (Vision-Language Multi-Dimensional Reward), a framework that dynamically decomposes evaluation into granular, interpretable dimensions. Instead of outputting a monolithic scalar, VL-MDR employs a visual-aware gating mechanism to identify relevant dimensions and adaptively weight them (e.g., Hallucination, Reasoning) for each specific input. To support this, we curate a dataset of 321k vision-language preference pairs annotated across 21 fine-grained dimensions. Extensive experiments show that VL-MDR consistently outperforms existing open-source reward models on benchmarks like VL-RewardBench. Furthermore, we show that VL-MDR-constructed preference pairs effectively enable DPO alignment to mitigate visual hallucinations and improve reliability, providing a scalable solution for VLM alignment.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Diffusion and multimodal model alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 17 of its 22 references that are in the map. First posted on arXiv 7 Apr 2026.

Builds on

17 of 22 references.

Closest ancestors · 8

Same direction, newest first

  1. 2025Skywork-VL Reward: An Effective Reward Model for Multimodal Understanding and Reasoning12 May
  2. 2025VLRMBench: A Comprehensive and Challenging Benchmark for Vision-Language Reward Models10 Mar
  3. 2025MM-RLHF: The Next Step Forward in Multimodal LLM Alignment14 Feb
  4. 2025InternLM-XComposer2.5-Reward: A Simple Yet Effective Multi-Modal Reward Model21 Jan
  5. 2024RLAIF-V: Open-Source AI Feedback Leads to Super GPT-4V Trustworthiness27 May
  6. 2023Silkie: Preference Distillation for Large Visual Language Models17 Dec
  7. 2023RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedback1 Dec
  8. 2023Aligning Large Multimodal Models with Factually Augmented RLHF25 Sep

Other references · 9

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2025Qwen3 Technical Reportcited by 8,195
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  4. 2017Deep reinforcement learning from human preferencescited by 3,846
  5. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  6. 2024Prometheus-Vision: Vision-Language Model as a Judge for Fine-Grained Evaluationcited by 123
  7. 2024WildVision: Evaluating Vision-Language Models in the Wild with Human Preferencescited by 76
  8. 2025LLaVA-Critic-R1: Your Critic Model is Secretly a Strong Policy Modelcited by 34
  9. 2024VisionArena: 230K Real World User-VLM Conversations with Preference Labelscited by 22

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025VL-GenRM: Enhancing Vision-Language Verification via Vision Experts and Iterative Training
  2. 2024VL-RewardBench: A Challenging Benchmark for Vision-Language Generative Reward Models
  3. 2025Revisiting the Learning Objectives of Vision-Language Reward Models
  4. 2025Token-Level Inference-Time Alignment for Vision-Language Models
  5. 2024RoVRM: A Robust Visual Reward Model Optimized via Auxiliary Textual Preference Data
  6. 2024Systematic Reward Gap Optimization for Mitigating VLM Hallucinations
  7. 2025Vision-R1: Evolving Human-Free Alignment in Large Vision-Language Models via Vision-Guided Reinforcement Learning
  8. 2025ViLBench: A Suite for Vision-Language Process Reward Modeling