mlsanityData to 7 Oct 2026

DT2IT-MRM: Debiased Preference Construction and Iterative Training for Multimodal Reward Modeling

arXiv 2604.19544 ↗First posted 21 Apr 2026cs.AIcited by 0

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

also in 1 more
Abstract (arXiv)

Multimodal reward models (MRMs) play a crucial role in aligning Multimodal Large Language Models (MLLMs) with human preferences. Training a good MRM requires high-quality multimodal preference data. However, existing preference datasets face three key challenges: lack of granularity in preference strength, textual style bias, and unreliable preference signals. Besides, existing open-source multimodal preference datasets suffer from substantial noise, yet there is a lack of effective and scalable curation methods to enhance their quality. To address these limitations, we propose DT2IT-MRM, which integrates a Debiased preference construction pipeline, a novel reformulation of text-to-image (T2I) preference data, and an Iterative Training framework that curates existing multimodal preference datasets for Multimodal Reward Modeling. Our experimental results show that DT2IT-MRM achieves new state-of-the-art overall performance on three major benchmarks: VL-RewardBench, Multimodal RewardBench, and MM-RLHF-RewardBench.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 23 of its 67 references that are in the map. First posted on arXiv 21 Apr 2026.

Builds on

23 of 67 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025Qwen3-VL Technical Report26 Nov
  2. 2025BaseReward: A Strong Baseline for Multimodal Reward Model19 Sep
  3. 2025Evaluating MLLMs with Multimodal Multi-image Reasoning Benchmark4 Jun
  4. 2025Multimodal RewardBench: Holistic Evaluation of Reward Models for Vision Language Models20 Feb
  5. 2024Enhancing the Reasoning Ability of Multimodal Large Language Models via Mixed Preference Optimization15 Nov

Other references · 18

  1. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
  2. 2025Qwen2.5-VL Technical Reportcited by 5,418
  3. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,928
  4. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,769
  5. 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,419
  6. 2023Human Preference Score v2: A Solid Benchmark for Evaluating Human Preferences of Text-to-Image Synthesiscited by 847
  7. 2025MM-Eureka: Exploring the Frontiers of Multimodal Reasoning with Rule-based Reinforcement Learningcited by 294
  8. 2023G-LLaVA: Solving Geometric Problem with Multi-Modal Large Language Modelcited by 208
  9. 2025Skywork-Reward-V2: Scaling Preference Data Curation via Human-AI Synergycited by 204
  10. 2024Aligning Modalities in Vision Large Language Models via Preference Fine-tuningcited by 197
  11. 2025Unified Reward Model for Multimodal Understanding and Generationcited by 194
  12. 2024Secrets of RLHF in Large Language Models Part II: Reward Modelingcited by 170
6 more
  1. 2025VisualPRM: An Effective Process Reward Model for Multimodal Reasoningcited by 134
  2. 2025R1-Reward: Training Multimodal Reward Model Through Stable Reinforcement Learningcited by 65
  3. 2025LLaVA-Critic-R1: Your Critic Model is Secretly a Strong Policy Modelcited by 34
  4. 2025Skywork-VL Reward: An Effective Reward Model for Multimodal Understanding and Reasoningcited by 30
  5. 2025Generative RLHF-V: Learning Principles from Multi-modal Human Preferencecited by 13
  6. 2025Omni-Reward: Towards Generalist Omni-Modal Reward Modeling with Free-Form Preferencescited by 11

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025The Devil Is in the Details: Tackling Unimodal Spurious Correlations for Generalizable Multimodal Reward Models
  2. 2025Multimodal RewardBench 2: Evaluating Omni Reward Models for Interleaved Text and Image
  3. 2026Diffusion Reward Models
  4. 2025From Captions to Rewards (CAREVL): Leveraging Large Language Model Experts for Enhanced Reward Modeling in Large Vision-Language Models
  5. 2025Multimodal Large Language Models with Adaptive Preference Optimization for Sequential Recommendation
  6. 2024Strengthening Multimodal Large Language Model with Bootstrapped Preference Optimization
  7. 2025Limited Preference Data? Learning Better Reward Model with Latent Space Synthesis
  8. 2026Entropy-Guided Data-Efficient Training for Multimodal Reasoning Reward Models