# Improving Generalization Robustness of Multimodal RLVR

arXiv: https://arxiv.org/abs/2608.08802 · HTML: https://mlsanity.com/paper/2608.08802/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 32 of its 39 references that are in the map.
- First posted on arXiv 9 Aug 2026.

## Placed in

- Multimodal LLMs / vision-language models / [Embodied and 3D vision-language models](https://mlsanity.com/mllm/embodied-and-3d-vision-language-models/)

## Builds on (most cited first, 30 of 32)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Qwen2.5-VL Technical Report](https://mlsanity.com/paper/2502.13923/) (2025)
- [Qwen3-VL Technical Report](https://mlsanity.com/paper/2511.21631/) (2025)
- [MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts](https://mlsanity.com/paper/2310.02255/) (2023)
- [Understanding R1-Zero-Like Training: A Critical Perspective](https://mlsanity.com/paper/2503.20783/) (2025)
- [DocVQA: A Dataset for VQA on Document Images](https://mlsanity.com/paper/2007.00398/) (2020)
- [Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalization](https://mlsanity.com/paper/1911.08731/) (2019)
- [OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problems](https://mlsanity.com/paper/2402.14008/) (2024)
- [Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?](https://mlsanity.com/paper/2504.13837/) (2025)
- [PathVQA: 30000+ Questions for Medical Visual Question Answering](https://mlsanity.com/paper/2003.10286/) (2020)
- [e-SNLI: Natural Language Inference with Natural Language Explanations](https://mlsanity.com/paper/1812.01193/) (2018)
- [MM-Eureka: Exploring the Frontiers of Multimodal Reasoning with Rule-based Reinforcement Learning](https://mlsanity.com/paper/2503.07365/) (2025)
- [REINFORCE++: Stabilizing Critic-Free Policy Optimization with Global Advantage Normalization](https://mlsanity.com/paper/2501.03262/) (2025)
- [Lingshu: A Generalist Foundation Model for Unified Multimodal Medical Understanding and Reasoning](https://mlsanity.com/paper/2506.07044/) (2025)
- [A Survey of Reinforcement Learning for Large Reasoning Models](https://mlsanity.com/paper/2509.08827/) (2025)
- [Document Understanding Dataset and Evaluation (DUDE)](https://mlsanity.com/paper/2305.08455/) (2023)
- [GMAI-MMBench: A Comprehensive Multimodal Evaluation Benchmark Towards General Medical AI](https://mlsanity.com/paper/2408.03361/) (2024)
- [On the Value of Out-of-Distribution Testing: An Example of Goodhart's Law](https://mlsanity.com/paper/2005.09241/) (2020)
- [When LLM Meets DRL: Advancing Jailbreaking Efficiency via DRL-guided Search](https://mlsanity.com/paper/2406.08705/) (2024)
- [RLHFPoison: Reward Poisoning Attack for Reinforcement Learning with Human Feedback in Large Language Models](https://mlsanity.com/paper/2311.09641/) (2023)
- [OpenING: A Comprehensive Benchmark for Judging Open-ended Interleaved Image-Text Generation](https://mlsanity.com/paper/2411.18499/) (2024)
- [Dynamic Multimodal Evaluation with Flexible Complexity by Vision-Language Bootstrapping](https://mlsanity.com/paper/2410.08695/) (2024)
- [MDK12-Bench: A Multi-Discipline Benchmark for Evaluating Reasoning in Multimodal Large Language Models](https://mlsanity.com/paper/2504.05782/) (2025)
- [Agent-as-a-Router: Agentic Model Routing for Coding Tasks](https://mlsanity.com/paper/2606.22902/) (2026)
- [R-Horizon: How Far Can Your Large Reasoning Model Really Go in Breadth and Depth?](https://mlsanity.com/paper/2510.08189/) (2025)
- [Adversarial Preference Learning for Robust LLM Alignment](https://mlsanity.com/paper/2505.24369/) (2025)
- [Information-Theoretic Reward Decomposition for Generalizable RLHF](https://mlsanity.com/paper/2504.06020/) (2025)
- [Toward Evaluative Thinking: Meta Policy Optimization with Evolving Reward Models](https://mlsanity.com/paper/2504.20157/) (2025)
- [Causally-Enhanced Reinforcement Policy Optimization](https://mlsanity.com/paper/2509.23095/) (2025)

## Abstract (arXiv)

Reinforcement Learning with Verifiable Rewards (RLVR) makes Multimodal Large Language Models more accurate, but the gains are brittle: simply paraphrasing a question or changing the prompt template can degrade them, which challenges reliable deployment in high-stakes scenarios like medical VQA. We trace this to two issues of the standard RL objective. First, the binary verifier conflates format with content, so the reward signal cannot tell a wrong answer apart from a misformatted one. Second, the training distribution covers only a thin slice of the real-world prompts that the model might meet at deployment, so policies that perform well on the training distribution can behave differently under unseen prompts during test. Both failures call for a robust post-training method that helps the policy cover a broader distribution of semantically equivalent prompts, and we identify two measures that help achieve this objective: separating format from semantics in the reward, and applying policy invariance across perturbed prompts with equivalent semantics. We therefore propose Prompt-Invariant RLVR (PIRL), consisting of a dynamic trinary reward and a consistency regularizer based on an embedding-space adversary. Under stress testing, PIRL's average accuracy on benchmarks drops by only ≤ 1%, where GRPO drops ~3%. On dynamic evaluation, PIRL also achieves the smallest performance drop.