# Not All Tokens See Equally: Perception-Grounded Policy Optimization for Large Vision-Language Models

arXiv: https://arxiv.org/abs/2604.01840 · HTML: https://mlsanity.com/paper/2604.01840/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal reasoning, one of 17 directions in LLM reasoning.
- Builds on 29 of its 56 references that are in the map.
- First posted on arXiv 2 Apr 2026.

## Placed in

- LLM reasoning / [Multimodal reasoning](https://mlsanity.com/lr/multimodal-reasoning/)
- Multimodal LLMs / vision-language models / [Embodied and 3D vision-language models](https://mlsanity.com/mllm/embodied-and-3d-vision-language-models/)

## Builds on (most cited first, 29 of 29)

- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [MMBench: Is Your Multi-modal Model an All-around Player?](https://mlsanity.com/paper/2307.06281/) (2023)
- [HybridFlow: A Flexible and Efficient RLHF Framework](https://mlsanity.com/paper/2409.19256/) (2024)
- [Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding](https://mlsanity.com/paper/2311.16922/) (2023)
- [Object Hallucination in Image Captioning](https://mlsanity.com/paper/1809.02156/) (2018)
- [Measuring Multimodal Mathematical Reasoning with MATH-Vision Dataset](https://mlsanity.com/paper/2402.14804/) (2024)
- [Inter-GPS: Interpretable Geometry Problem Solving with Formal Language and Symbolic Reasoning](https://mlsanity.com/paper/2105.04165/) (2021)
- [Absolute Zero: Reinforced Self-play Reasoning with Zero Data](https://mlsanity.com/paper/2505.03335/) (2025)
- [VL-Rethinker: Incentivizing Self-Reflection of Vision-Language Models with Reinforcement Learning](https://mlsanity.com/paper/2504.08837/) (2025)
- [LogicVista: Multimodal LLM Logical Reasoning Benchmark in Visual Contexts](https://mlsanity.com/paper/2407.04973/) (2024)
- [HALC: Object Hallucination Reduction via Adaptive Focal-Contrast Decoding](https://mlsanity.com/paper/2403.00425/) (2024)
- [Insight-V: Exploring Long-Chain Visual Reasoning with Multimodal Large Language Models](https://mlsanity.com/paper/2411.14432/) (2024)
- [Perception-Aware Policy Optimization for Multimodal Reasoning](https://mlsanity.com/paper/2507.06448/) (2025)
- [Visionary-R1: Mitigating Shortcuts in Visual Reasoning with Reinforcement Learning](https://mlsanity.com/paper/2505.14677/) (2025)
- [NoisyRollout: Reinforcing Visual Reasoning with Data Augmentation](https://mlsanity.com/paper/2504.13055/) (2025)
- [UniVG-R1: Reasoning Guided Universal Visual Grounding with Reinforcement Learning](https://mlsanity.com/paper/2505.14231/) (2025)
- [Reinforcing Multi-Turn Reasoning in LLM Agents via Fine-Grained Reward Structure and Credit Assignment](https://mlsanity.com/paper/2505.11821/) (2025)
- [Advancing Multimodal Reasoning: From Optimized Cold Start to Staged Reinforcement Learning](https://mlsanity.com/paper/2506.04207/) (2025)
- [Spotlight on Token Perception for Multimodal Reinforcement Learning](https://mlsanity.com/paper/2510.09285/) (2025)
- [Unlocking Exploration in RLVR: Uncertainty-aware Advantage Shaping for Deeper Reasoning](https://mlsanity.com/paper/2510.10649/) (2025)
- [One RL to See Them All: Visual Triple Unified Reinforcement Learning](https://mlsanity.com/paper/2505.18129/) (2025)
- [MoDoMoDo: Multi-Domain Data Mixtures for Multimodal LLM Reinforcement Learning](https://mlsanity.com/paper/2505.24871/) (2025)
- [Attention Illuminates LLM Reasoning: The Preplan-and-Anchor Rhythm Enables Fine-Grained Policy Optimization](https://mlsanity.com/paper/2510.13554/) (2025)
- [Causal Tracing of Object Representations in Large Vision Language Models: Mechanistic Interpretability and Hallucination Mitigation](https://mlsanity.com/paper/2511.05923/) (2025)
- [An Information-Theoretic Perspective on Credit Assignment in Reinforcement Learning](https://mlsanity.com/paper/2103.06224/) (2021)
- [Truth in the Few: High-Value Data Selection for Efficient Multi-Modal Reasoning](https://mlsanity.com/paper/2506.04755/) (2025)
- [CLAIM: Mitigating Multilingual Object Hallucination in Large Vision-Language Models with Cross-Lingual Attention Intervention](https://mlsanity.com/paper/2506.11073/) (2025)
- [MPR-GUI: Benchmarking and Enhancing Multilingual Perception and Reasoning in GUI Agents](https://mlsanity.com/paper/2512.00756/) (2025)

## Built on by (most cited first, 7 of 7)

- [Structured Role-Aware Policy Optimization for Multimodal Reasoning](https://mlsanity.com/paper/2605.07274/) (2026)
- [ELSA3D: Elastic Semantic Anchoring for Unified 3D Understanding and Generation](https://mlsanity.com/paper/2607.06565/) (2026)
- [Entropy Is Not Enough: Unlocking Effective Reinforcement Learning for Visual Reasoning via Vision-Anchored Token Selection](https://mlsanity.com/paper/2606.03937/) (2026)
- [Anchoring What Matters: A Dual-Level Learning Framework for Visually-Grounded Multimodal Reasoning](https://mlsanity.com/paper/2609.18057/) (2026)
- [TTRSD: Test-Time Reinforcement Learning with Self-Distillation for Vision-Language Models](https://mlsanity.com/paper/2609.33414/) (2026)
- [Beyond Entropy: Self-Diagnostic Multi-Role Token Optimization for Video Reasoning](https://mlsanity.com/paper/2610.03400/) (2026)
- [Residual Visual Credit Optimization: Conserved Evidence Routing for Multimodal Reinforcement Learning](https://mlsanity.com/paper/2610.04918/) (2026)

## Abstract (arXiv)

While Reinforcement Learning from Verifiable Rewards (RLVR) has advanced reasoning in Large Vision-Language Models (LVLMs), prevailing frameworks suffer from a foundational methodological flaw: by distributing identical advantages across all generated tokens, these methods inherently dilute the learning signals essential for optimizing the critical, visually-grounded steps of multimodal reasoning. To bridge this gap, we formulate textitToken Visual Dependency, quantifying the causal information gain of visual inputs via the Kullback-Leibler (KL) divergence between visual-conditioned and text-only predictive distributions. Revealing that this dependency is highly sparse and semantically pivotal, we introduce Perception-Grounded Policy Optimization (PGPO), which is a novel fine-grained credit assignment framework that dynamically reshapes advantages at the token level. Through a threshold-gated, mass-conserving mechanism, PGPO actively amplifies learning signals for visually-dependent tokens while suppressing gradient noise from linguistic priors. Extensive experiments based on the Qwen2.5-VL series across seven challenging multimodal reasoning benchmarks demonstrate that PGPO boosts models by 18.7% on average. Both theoretical and empirical analyses confirm that PGPO effectively reduces gradient variance, prevents training collapse, and acts as a potent regularizer for robust, perception-grounded multimodal reasoning. Code will be released on https://github.com/Yzk1114/PGPO.