When Does a Video-Language Model Stop Watching? Reward Strength Controls the Formation and Reversal of Visual Shortcuts in Multimodal RLVR
On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models
Abstract (arXiv)
Reinforcement learning with verifiable rewards (RLVR) is increasingly applied to large vision-language models (LVLMs), yet outcome-only optimization can drive a model to stop attending to the video and instead exploit linguistic priors -- a failure we call a visual shortcut. While the existence of such perception bypass is by now documented, how it forms, whether it can be undone, and when intervention still helps remain open. We treat the strength of a grounding penalty, lambda, as a control knob and characterize the formation-reversal dynamics of visual shortcuts along the training time axis. On a held-out, out-of-distribution diagnostic set, we find: (i) a sharp onset -- shortcut reliance emerges abruptly over a narrow window of optimization steps and is robust across random seeds; (ii) a monotone dose-response -- increasing lambda progressively suppresses the shortcut, and at an intermediate dose the trajectory first forms and then reverses the shortcut, exposing a hysteresis-like asymmetry between acquiring and removing it; and (iii) a critical intervention window -- applying the penalty before onset arrests shortcut formation, whereas the same penalty applied after consolidation is markedly less effective. Together these results recast visual-shortcut collapse not as a binary defect but as a controllable, time-dependent, and asymmetric process, with direct implications for when and how strongly to regularize multimodal RLVR.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 13 of its 14 references that are in the map. First posted on arXiv 20 Jun 2026.
Builds on
13 of 14 references.
References · 13
- 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
- 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,109
- 2024BLINK: Multimodal Large Language Models Can See but Not Perceivecited by 550
- 2026Reward Hacking in the Era of Large Models: Mechanisms, Emergent Misalignment, Challengescited by 33
- 2026LLMs Gaming Verifiers: RLVR can Lead to Reward Hackingcited by 28
- 2026Countdown-Code: A Testbed for Studying The Emergence and Generalization of Reward Hacking in RLVRcited by 17
- 2025Understanding Language Prior of LVLMs by Contrasting Chain-of-Embeddingcited by 14
- 2026Spurious Rewards Paradox: Mechanistically Understanding How RLVR Activates Memorization Shortcuts in LLMscited by 14
- 2026VLMs Need Words: Vision Language Models Ignore Visual Detail In Favor of Semantic Anchorscited by 13
- 2026Adversarial Reward Auditing for Active Detection and Mitigation of Reward Hackingcited by 11
- 2026Do Vision-Language Models See or Guess? Measuring and Reducing Textual-Prior Reliance with a Phrasing-Controlled Benchmarkcited by 8
- 2026From Rebound to Remedy: Understanding and Mitigating Reward Hacking via Representation Engineeringcited by 7
Built on by
Citing papers per year: 2026: 2.
2 papers.
Similar, not cited
Close in wording, no citation either way
- 2026Same Reward, Different Skills: When Multimodal RL Learns to Look
- 2026Video Models Can Reason with Verifiable Rewards
- 2026MIRL: Mutual Information-Guided Reinforcement Learning for Vision-Language Models
- 2026EasyVideoR1: Easier RL for Video Understanding
- 2025Seeing to Act, Prompting to Specify: A Bayesian Factorization of Vision Language Action Policy
- 2026When Listening Becomes Easier: Scrubbing Visual Cues for Shortcut-Free VLAs
- 2026When Instructions Retrieve Trajectories: Diagnosing and Mitigating Generalization Failures in VLA Models
- 2025Rethinking RL Scaling for Vision Language Models: A Transparent, From-Scratch Framework and Comprehensive Evaluation Scheme