mlsanityData to 14 Oct 2026

The Devil Is in the Details: Tackling Unimodal Spurious Correlations for Generalizable Multimodal Reward Models

arXiv 2503.03122 ↗First posted 5 Mar 2025cs.CL cs.AIcited by 2

On the map: LLM alignment / RLHF / preference optimisation → Diffusion and multimodal model alignment

Abstract (arXiv)

Multimodal Reward Models (MM-RMs) are crucial for aligning Large Language Models (LLMs) with human preferences, particularly as LLMs increasingly interact with multimodal data. However, we find that MM-RMs trained on existing datasets often struggle to generalize to out-of-distribution data due to their reliance on unimodal spurious correlations, primarily text-only shortcuts within the training distribution, which prevents them from leveraging true multimodal reward functions. To address this, we introduce a Shortcut-aware MM-RM learning algorithm that mitigates this issue by dynamically reweighting training samples, shifting the distribution toward better multimodal understanding, and reducing dependence on unimodal spurious correlations. Our experiments demonstrate significant improvements in generalization, downstream task performance, and scalability, establishing a more robust framework for multimodal reward modeling.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Diffusion and multimodal model alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 39 of its 63 references that are in the map. First posted on arXiv 5 Mar 2025.

Builds on

39 of 63 references.

Closest ancestors · 6

Same direction, newest first

  1. 2025InternLM-XComposer2.5-Reward: A Simple Yet Effective Multi-Modal Reward Model21 Jan
  2. 2024VL-RewardBench: A Challenging Benchmark for Vision-Language Generative Reward Models26 Nov
  3. 2024RoVRM: A Robust Visual Reward Model Optimized via Auxiliary Textual Preference Data22 Aug
  4. 2024Understanding Alignment in Multimodal LLMs: A Comprehensive Study2 Jul
  5. 2024Aligning Modalities in Vision Large Language Models via Preference Fine-tuning18 Feb
  6. 2023Aligning Large Multimodal Models with Factually Augmented RLHF25 Sep

Other references · 33

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  5. 2023Qwen Technical Reportcited by 3,344
  6. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  7. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,986
  8. 2019Invariant Risk Minimizationcited by 1,854
  9. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,745
  10. 2016Concrete Problems in AI Safetycited by 1,642
  11. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,599
  12. 2019Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalizationcited by 1,247
21 more
  1. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,239
  2. 2023A Survey on Multimodal Large Language Modelscited by 909
  3. 2024RewardBench: Evaluating Reward Models for Language Modelingcited by 499
  4. 2024Generative Verifiers: Reward Modeling as Next-Token Predictioncited by 480
  5. 2024Improve Mathematical Reasoning in Language Models by Automated Process Supervisioncited by 441
  6. 2022The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Modelscited by 328
  7. 2023Analyzing and Mitigating Object Hallucination in Large Vision-Language Modelscited by 316
  8. 2023A Survey of Reinforcement Learning from Human Feedbackcited by 264
  9. 2023Reward Model Ensembles Help Mitigate Overoptimizationcited by 231
  10. 2023SVIT: Scaling up Visual Instruction Tuningcited by 162
  11. 2024Regularizing Hidden States Enables Learning Generalizable Reward Model for LLMscited by 142
  12. 2024ODIN: Disentangled Reward Mitigates Hacking in RLHFcited by 136
  13. 2024ARGS: Alignment as Reward-Guided Searchcited by 103
  14. 2022Spread Spurious Attribute: Improving Worst-group Accuracy with Spurious Attribute Estimationcited by 88
  15. 2023Loose lips sink ships: Mitigating Length Bias in Reinforcement Learning from Human Feedbackcited by 78
  16. 2023On the Foundations of Shortcut Learningcited by 65
  17. 2020Empirical or Invariant Risk Minimization? A Sample Complexity Perspectivecited by 62
  18. 2024Two Effects, One Trigger: On the Modality Gap, Object Bias, and Information Imbalance in Contrastive Vision-Language Modelscited by 57
  19. 2023Reformulating Vision-Language Foundation Models and Datasets Towards Universal Multimodal Assistantscited by 25
  20. 2023Understanding Unimodal Bias in Multimodal Deep Linear Networkscited by 19
  21. 2024MetaRM: Shifted Distributions Alignment via Meta-Learningcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 1.

2 papers.

  1. 2025GM-PRM: A Generative Multimodal Process Reward Model for Multimodal Mathematical Reasoningcited by 14
  2. 2026Toward Native Multimodal Modeling: A Roadmapcited by 7

Similar, not cited

Close in wording, no citation either way

  1. 2025BaseReward: A Strong Baseline for Multimodal Reward Model
  2. 2026DT2IT-MRM: Debiased Preference Construction and Iterative Training for Multimodal Reward Modeling
  3. 2025Multimodal RewardBench 2: Evaluating Omni Reward Models for Interleaved Text and Image
  4. 2025MM-RLHF: The Next Step Forward in Multimodal LLM Alignment
  5. 2026Diffusion Reward Models
  6. 2026Entropy-Guided Data-Efficient Training for Multimodal Reasoning Reward Models
  7. 2024M-RewardBench: Evaluating Reward Models in Multilingual Settings
  8. 2026MSRL: Scaling Generative Multimodal Reward Modeling via Multi-Stage Reinforcement Learning