Not All Tokens See Equally: Perception-Grounded Policy Optimization for Large Vision-Language Models
On the map: LLM reasoning → Multimodal reasoning
Abstract (arXiv)
While Reinforcement Learning from Verifiable Rewards (RLVR) has advanced reasoning in Large Vision-Language Models (LVLMs), prevailing frameworks suffer from a foundational methodological flaw: by distributing identical advantages across all generated tokens, these methods inherently dilute the learning signals essential for optimizing the critical, visually-grounded steps of multimodal reasoning. To bridge this gap, we formulate textitToken Visual Dependency, quantifying the causal information gain of visual inputs via the Kullback-Leibler (KL) divergence between visual-conditioned and text-only predictive distributions. Revealing that this dependency is highly sparse and semantically pivotal, we introduce Perception-Grounded Policy Optimization (PGPO), which is a novel fine-grained credit assignment framework that dynamically reshapes advantages at the token level. Through a threshold-gated, mass-conserving mechanism, PGPO actively amplifies learning signals for visually-dependent tokens while suppressing gradient noise from linguistic priors. Extensive experiments based on the Qwen2.5-VL series across seven challenging multimodal reasoning benchmarks demonstrate that PGPO boosts models by 18.7% on average. Both theoretical and empirical analyses confirm that PGPO effectively reduces gradient variance, prevents training collapse, and acts as a potent regularizer for robust, perception-grounded multimodal reasoning. Code will be released on https://github.com/Yzk1114/PGPO.
Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 29 of its 56 references that are in the map. First posted on arXiv 2 Apr 2026.
Builds on
29 of 56 references.
Closest ancestors · 13
Same direction, newest first
- 2025MPR-GUI: Benchmarking and Enhancing Multilingual Perception and Reasoning in GUI Agents30 Nov
- 2025Spotlight on Token Perception for Multimodal Reinforcement Learning10 Oct
- 2025Perception-Aware Policy Optimization for Multimodal Reasoning8 Jul
- 2025Truth in the Few: High-Value Data Selection for Efficient Multi-Modal Reasoning5 Jun
- 2025Advancing Multimodal Reasoning: From Optimized Cold Start to Staged Reinforcement Learning4 Jun
- 2025Visionary-R1: Mitigating Shortcuts in Visual Reasoning with Reinforcement Learning20 May
- 2025UniVG-R1: Reasoning Guided Universal Visual Grounding with Reinforcement Learning20 May
- 2025NoisyRollout: Reinforcing Visual Reasoning with Data Augmentation17 Apr
- 2025VL-Rethinker: Incentivizing Self-Reflection of Vision-Language Models with Reinforcement Learning10 Apr
- 2024Insight-V: Exploring Long-Chain Visual Reasoning with Multimodal Large Language Models21 Nov
- 2024LogicVista: Multimodal LLM Logical Reasoning Benchmark in Visual Contexts6 Jul
- 2024Measuring Multimodal Mathematical Reasoning with MATH-Vision Dataset22 Feb
1 more
Other references · 16
- 2014Microsoft COCO: Common Objects in Contextcited by 17,498
- 2017Proximal Policy Optimization Algorithmscited by 14,441
- 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 1,980
- 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decodingcited by 671
- 2018Object Hallucination in Image Captioningcited by 661
- 2021Inter-GPS: Interpretable Geometry Problem Solving with Formal Language and Symbolic Reasoningcited by 376
- 2025Absolute Zero: Reinforced Self-play Reasoning with Zero Datacited by 323
- 2024HALC: Object Hallucination Reduction via Adaptive Focal-Contrast Decodingcited by 171
- 2025Reinforcing Multi-Turn Reasoning in LLM Agents via Fine-Grained Reward Structure and Credit Assignmentcited by 77
- 2025Unlocking Exploration in RLVR: Uncertainty-aware Advantage Shaping for Deeper Reasoningcited by 34
- 2025One RL to See Them All: Visual Triple Unified Reinforcement Learningcited by 29
- 2025MoDoMoDo: Multi-Domain Data Mixtures for Multimodal LLM Reinforcement Learningcited by 29
4 more
- 2025Attention Illuminates LLM Reasoning: The Preplan-and-Anchor Rhythm Enables Fine-Grained Policy Optimizationcited by 23
- 2025Causal Tracing of Object Representations in Large Vision Language Models: Mechanistic Interpretability and Hallucination Mitigationcited by 22
- 2021An Information-Theoretic Perspective on Credit Assignment in Reinforcement Learningcited by 20
- 2025CLAIM: Mitigating Multilingual Object Hallucination in Large Vision-Language Models with Cross-Lingual Attention Interventioncited by 11
Built on by
Citing papers per year: 2026: 5.
5 papers.
- 2026Structured Role-Aware Policy Optimization for Multimodal Reasoningcited by 1
- 2026ELSA3D: Elastic Semantic Anchoring for Unified 3D Understanding and Generationcited by 1
- 2026Entropy Is Not Enough: Unlocking Effective Reinforcement Learning for Visual Reasoning via Vision-Anchored Token Selectioncited by 0
- 2026Anchoring What Matters: A Dual-Level Learning Framework for Visually-Grounded Multimodal Reasoningcited by 0
- 2026TTRSD: Test-Time Reinforcement Learning with Self-Distillation for Vision-Language Modelscited by 0
- 2026TTRSD: Test-Time Reinforcement Learning with Self-Distillation for Vision-Language Models27 Sep 2026
- 2026Anchoring What Matters: A Dual-Level Learning Framework for Visually-Grounded Multimodal Reasoning16 Sep 2026
- 2026ELSA3D: Elastic Semantic Anchoring for Unified 3D Understanding and Generation7 Jul 2026
- 2026Entropy Is Not Enough: Unlocking Effective Reinforcement Learning for Visual Reasoning via Vision-Anchored Token Selection2 Jun 2026
- 2026Structured Role-Aware Policy Optimization for Multimodal Reasoning8 May 2026
Similar, not cited
Close in wording, no citation either way
- 2026PRPO: Perception-Reinforced Policy Optimization via Token-Level Dynamic Advantage Reshaping
- 2026Reinforcing Multimodal Reasoning via Token-Level Perception-Grounded Advantage Estimation
- 2026Rethinking Token-Level Policy Optimization for Multimodal Chain-of-Thought
- 2026Bridging Perception and Reasoning: Token Reweighting for RLVR in Multimodal LLMs
- 2026Visually-Guided Policy Optimization for Multimodal Reasoning
- 2026Thinking with Deltas: Incentivizing Reinforcement Learning via Differential Visual Reasoning Policy
- 2026PDCR: Perception-Decomposed Confidence Reward for Vision-Language Reasoning
- 2025More Thought, Less Accuracy? On the Dual Nature of Reasoning in Vision-Language Models