mlsanityData to 7 Oct 2026

Entropy-Guided Data-Efficient Training for Multimodal Reasoning Reward Models

arXiv 2602.01884 ↗First posted 2 Feb 2026cs.AI cs.LGcited by 1

On the map: LLM reasoning → Multimodal reasoning

also in 2 more
Abstract (arXiv)

Multimodal reward models are crucial for aligning multimodal large language models with human preferences. Recent works have incorporated reasoning capabilities into these models, achieving promising results. However, training these models suffers from two critical challenges: (1) the inherent noise in preference datasets, which degrades model performance, and (2) the inefficiency of conventional training methods, which ignore the differences in sample difficulty. In this paper, we identify a strong correlation between response entropy and accuracy, indicating that entropy can serve as a reliable and unsupervised proxy for annotation noise and sample difficulty. Based on this insight, we propose a novel Entropy-Guided Training (EGT) approach for multimodal reasoning reward models, which combines two strategies: (1) entropy-guided data curation to mitigate the impact of unreliable samples, and (2) an entropy-guided training strategy that progressively introduces more complex examples. Extensive experiments across three benchmarks show that the EGT-trained model consistently outperforms state-of-the-art multimodal reward models.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 25 of its 29 references that are in the map. First posted on arXiv 2 Feb 2026.

Builds on

25 of 29 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025InternLM-XComposer2.5-Reward: A Simple Yet Effective Multi-Modal Reward Model21 Jan
  2. 2024Benchmarking Large Multimodal Models against Common Corruptions22 Jan
  3. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul

Other references · 22

  1. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
  2. 2025Qwen2.5-VL Technical Reportcited by 5,418
  3. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  4. 2020Learning to summarize from human feedbackcited by 2,562
  5. 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,442
  6. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackcited by 615
  7. 2023RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedbackcited by 404
  8. 2023What Makes Good Data for Alignment? A Comprehensive Study of Automatic Data Selection in Instruction Tuningcited by 354
  9. 2024Aligning Modalities in Vision Large Language Models via Preference Fine-tuningcited by 197
  10. 2025VITA-1.5: Towards GPT-4o Level Real-Time Vision and Speech Interactioncited by 191
  11. 2024Secrets of RLHF in Large Language Models Part II: Reward Modelingcited by 170
  12. 2025RM-R1: Reward Modeling as Reasoningcited by 149
10 more
  1. 2024VL-RewardBench: A Challenging Benchmark for Vision-Language Generative Reward Modelscited by 85
  2. 2025MM-RLHF: The Next Step Forward in Multimodal LLM Alignmentcited by 79
  3. 2024WildVision: Evaluating Vision-Language Models in the Wild with Human Preferencescited by 76
  4. 2025R1-Reward: Training Multimodal Reward Model Through Stable Reinforcement Learningcited by 65
  5. 2024VLFeedback: A Large-Scale AI Feedback Dataset for Large Vision-Language Models Alignmentcited by 56
  6. 2024Impact of Preference Noise on the Alignment Performance of Generative Language Modelscited by 41
  7. 2025Multimodal RewardBench: Holistic Evaluation of Reward Models for Vision Language Modelscited by 40
  8. 2024Curry-DPO: Enhancing Alignment using Curriculum Learning & Ranked Preferencescited by 39
  9. 2025More Bang for the Buck: Process Reward Modeling with Entropy-Driven Uncertaintycited by 10
  10. 2025Reinforcement Mid-Trainingcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026RISE: Reliable Improvement in Self-Evolving Vision-Language Modelscited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2025Multimodal RewardBench 2: Evaluating Omni Reward Models for Interleaved Text and Image
  2. 2025Revisiting Entropy in Reinforcement Learning for Large Reasoning Models
  3. 2025The Devil Is in the Details: Tackling Unimodal Spurious Correlations for Generalizable Multimodal Reward Models
  4. 2025BaseReward: A Strong Baseline for Multimodal Reward Model
  5. 2026GMTS: Gradient Magnitude-based Token Selection Improves RLVR Training for LLM Reasoning
  6. 2026AdaThinking-E: One-Token Entropy Regulation for Adaptive Thinking
  7. 2026Reinforcing Multimodal Reasoning via Token-Level Perception-Grounded Advantage Estimation
  8. 2025ARES: Multimodal Adaptive Reasoning via Difficulty-Aware Token-Level Entropy Shaping