mlsanityData to 14 Oct 2026

Multimodal Reward Hacking in Reinforcement Learning

arXiv 2607.09492 ↗First posted 10 Jul 2026cs.AIcited by 2

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

also in 2 more
Abstract (arXiv)

Reinforcement learning (RL) is increasingly used to align multimodal large language models (MLLMs), but higher rewards do not always imply better task performance. This risk is amplified when visual evidence is evaluated by text-only or weakly grounded rewards. We study reward hacking in MLLM RL across safety VQA, chart VQA, and stress-test settings, varying reward design, data ambiguity, model scale (2B-32B), and RL algorithm (GRPO, RLOO, DAPO). We introduce Newly Rewarded Failure Rate (NRFR), which measures failures among samples whose proxy reward improves over the SFT baseline. Outcome-only rewards cause severe hacking, reaching 48.1% Reward Hacking Rate (RHR), while NRFR exceeding RHR shows that RL creates new failures rather than merely inheriting them. Scaling reduces but does not eliminate hacking: even the 32B model retains a 54.9% worse rate under outcome-only rewards, whereas answer-aware rewards improve the oracle trend at every scale. Robustness is also algorithm- and scale-dependent: GRPO is consistently most resistant, RLOO remains vulnerable, and DAPO improves substantially from 2B to 8B. Visual-evidence rewards help only with reliable verification: keyword-based checks increase hacking, while VLM-as-judge semantic verification reduces it. Overall, multimodal reward hacking is a systematic result of optimizing imperfect rewards, and robust alignment requires rewards and verifiers that remain reliable under optimization pressure.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 41 of its 45 references that are in the map. First posted on arXiv 10 Jul 2026.

Builds on

41 of 45 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025Perceptual-Evidence Anchored Reinforced Learning for Multimodal Reasoning23 Nov
  2. 2025More Thought, Less Accuracy? On the Dual Nature of Reasoning in Vision-Language Models30 Sep
  3. 2025Generative RLHF-V: Learning Principles from Multi-modal Human Preference24 May
  4. 2025VLM-R1: A Stable and Generalizable R1-style Large Vision-Language Model10 Apr

Other references · 37

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2024GPT-4o System Cardcited by 4,588
  3. 2025Qwen3-VL Technical Reportcited by 2,793
  4. 2024WildGuard: Open One-Stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMscited by 496
  5. 2023MM-SafetyBench: A Benchmark for Safety Evaluation of Multimodal Large Language Modelscited by 274
  6. 2025Spurious Rewards: Rethinking Training Signals in RLVRcited by 209
  7. 2025Unified Reward Model for Multimodal Understanding and Generationcited by 199
  8. 2024Safety Fine-Tuning at (Almost) No Cost: A Baseline for Vision Large Language Modelscited by 167
  9. 2025Pref-GRPO: Pairwise Preference Reward-based GRPO for Stable Text-to-Image Reinforcement Learningcited by 92
  10. 2025Reward Shaping to Mitigate Reward Hacking in RLHFcited by 88
  11. 2024SPA-VL: A Comprehensive Safety Preference Alignment Dataset for Vision Language Modelcited by 85
  12. 2025MM-RLHF: The Next Step Forward in Multimodal LLM Alignmentcited by 83
25 more
  1. 2025On the Interplay of Pre-Training, Mid-Training, and RL on Reasoning Language Modelscited by 69
  2. 2024VLSBench: Unveiling Visual Leakage in Multimodal Safetycited by 63
  3. 2024MLLMGuard: A Multi-dimensional Safety Evaluation Suite for Multimodal Large Language Modelscited by 34
  4. 2026Reward Hacking in the Era of Large Models: Mechanisms, Emergent Misalignment, Challengescited by 33
  5. 2024Reward-Robust RLHF in LLMscited by 32
  6. 2025Uncalibrated Reasoning: GRPO Induces Overconfidence for Stochastic Outcomescited by 22
  7. 2025BaseReward: A Strong Baseline for Multimodal Reward Modelcited by 15
  8. 2025MSTS: A Multimodal Safety Test Suite for Vision-Language Modelscited by 12
  9. 2026F-GRPO: Don't Let Your Policy Learn the Obvious and Forget the Rarecited by 11
  10. 2025PerPO: Perceptual Preference Optimization via Discriminative Rewardingcited by 9
  11. 2025VLSU: Mapping the Limits of Joint Multimodal Understanding for AI Safetycited by 8
  12. 2026Gradient Regularization Mitigates Reward Hacking in Reinforcement Learning from Human Feedback and Verifiable Rewardscited by 8
  13. 2026Demystifying Reinforcement Learning for Long-Horizon Tool-Using Agents: A Comprehensive Recipecited by 8
  14. 2025Activation Reward Models for Few-Shot Model Alignmentcited by 6
  15. 2025DUAL-Bench: Measuring Over-Refusal and Robustness in Vision-Language Modelscited by 6
  16. 2026Mitigating Reward Hacking in RLHF via Bayesian Non-negative Reward Modelingcited by 5
  17. 2026Reward Hacking as Equilibrium under Finite Evaluationcited by 5
  18. 2026Asymmetric Prompt Weighting for Reinforcement Learning with Verifiable Rewardscited by 4
  19. 2026Explaining and Preventing Alignment Collapse in Iterative RLHFcited by 4
  20. 2026Towards Understanding Specification Gaming in Reasoning Modelscited by 3
  21. 2026Balanced Aggregation: Understanding and Fixing Aggregation Bias in GRPOcited by 3
  22. 2026Do Post-Training Algorithms Actually Differ? A Controlled Study Across Model Scales Uncovers Scale-Dependent Ranking Inversionscited by 2
  23. 2026Robust Optimization for Mitigating Reward Hacking with Correlated Proxiescited by 2
  24. 2026When Errors Can Be Beneficial: A Categorization of Imperfect Rewards for Policy Gradientcited by 1
  25. 2026Beyond Negative Rollouts: Positive-Only Policy Optimization with Implicit Negative Gradientscited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Prism-GRPO: Faster VLA Policy Optimization via Splitting Same-outcome Groupscited by 2
  2. 2026When the Judge Acts: Auditing VLM-Guided Image Selection on Culturally Situated Promptscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Countdown-Code: A Testbed for Studying The Emergence and Generalization of Reward Hacking in RLVR
  2. 2026Understanding Reward Hacking in Text-to-Image Reinforcement Learning
  3. 2026Reward Hacking Benchmark: Measuring Exploits in LLM Agents with Tool Use
  4. 2026CATCH: A Controllable Analysis Testbed for Reward Hacking in Coding RL
  5. 2026On the effectiveness of reward functions in reinforcement learning for confidence calibration of large language models
  6. 2026Directional Alignment Mitigates Reward Hacking in Reinforcement Learning for Language Models
  7. 2026Reward Modeling for Reinforcement Learning-Based LLM Reasoning: Design, Challenges, and Evaluation
  8. 2026Reward Hacking in Language Model Agents: Revisiting AI Safety Gridworlds