mlsanityData to 7 Oct 2026

Not All Tokens See Equally: Perception-Grounded Policy Optimization for Large Vision-Language Models

arXiv 2604.01840 ↗First posted 2 Apr 2026cs.AIcited by 5

On the map: LLM reasoning → Multimodal reasoning

also in 1 more
Abstract (arXiv)

While Reinforcement Learning from Verifiable Rewards (RLVR) has advanced reasoning in Large Vision-Language Models (LVLMs), prevailing frameworks suffer from a foundational methodological flaw: by distributing identical advantages across all generated tokens, these methods inherently dilute the learning signals essential for optimizing the critical, visually-grounded steps of multimodal reasoning. To bridge this gap, we formulate textitToken Visual Dependency, quantifying the causal information gain of visual inputs via the Kullback-Leibler (KL) divergence between visual-conditioned and text-only predictive distributions. Revealing that this dependency is highly sparse and semantically pivotal, we introduce Perception-Grounded Policy Optimization (PGPO), which is a novel fine-grained credit assignment framework that dynamically reshapes advantages at the token level. Through a threshold-gated, mass-conserving mechanism, PGPO actively amplifies learning signals for visually-dependent tokens while suppressing gradient noise from linguistic priors. Extensive experiments based on the Qwen2.5-VL series across seven challenging multimodal reasoning benchmarks demonstrate that PGPO boosts models by 18.7% on average. Both theoretical and empirical analyses confirm that PGPO effectively reduces gradient variance, prevents training collapse, and acts as a potent regularizer for robust, perception-grounded multimodal reasoning. Code will be released on https://github.com/Yzk1114/PGPO.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 29 of its 56 references that are in the map. First posted on arXiv 2 Apr 2026.

Builds on

29 of 56 references.

Closest ancestors · 13

Same direction, newest first

  1. 2025MPR-GUI: Benchmarking and Enhancing Multilingual Perception and Reasoning in GUI Agents30 Nov
  2. 2025Spotlight on Token Perception for Multimodal Reinforcement Learning10 Oct
  3. 2025Perception-Aware Policy Optimization for Multimodal Reasoning8 Jul
  4. 2025Truth in the Few: High-Value Data Selection for Efficient Multi-Modal Reasoning5 Jun
  5. 2025Advancing Multimodal Reasoning: From Optimized Cold Start to Staged Reinforcement Learning4 Jun
  6. 2025Visionary-R1: Mitigating Shortcuts in Visual Reasoning with Reinforcement Learning20 May
  7. 2025UniVG-R1: Reasoning Guided Universal Visual Grounding with Reinforcement Learning20 May
  8. 2025NoisyRollout: Reinforcing Visual Reasoning with Data Augmentation17 Apr
  9. 2025VL-Rethinker: Incentivizing Self-Reflection of Vision-Language Models with Reinforcement Learning10 Apr
  10. 2024Insight-V: Exploring Long-Chain Visual Reasoning with Multimodal Large Language Models21 Nov
  11. 2024LogicVista: Multimodal LLM Logical Reasoning Benchmark in Visual Contexts6 Jul
  12. 2024Measuring Multimodal Mathematical Reasoning with MATH-Vision Dataset22 Feb
1 more
  1. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul

Other references · 16

  1. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 1,980
  4. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decodingcited by 671
  5. 2018Object Hallucination in Image Captioningcited by 661
  6. 2021Inter-GPS: Interpretable Geometry Problem Solving with Formal Language and Symbolic Reasoningcited by 376
  7. 2025Absolute Zero: Reinforced Self-play Reasoning with Zero Datacited by 323
  8. 2024HALC: Object Hallucination Reduction via Adaptive Focal-Contrast Decodingcited by 171
  9. 2025Reinforcing Multi-Turn Reasoning in LLM Agents via Fine-Grained Reward Structure and Credit Assignmentcited by 77
  10. 2025Unlocking Exploration in RLVR: Uncertainty-aware Advantage Shaping for Deeper Reasoningcited by 34
  11. 2025One RL to See Them All: Visual Triple Unified Reinforcement Learningcited by 29
  12. 2025MoDoMoDo: Multi-Domain Data Mixtures for Multimodal LLM Reinforcement Learningcited by 29
4 more
  1. 2025Attention Illuminates LLM Reasoning: The Preplan-and-Anchor Rhythm Enables Fine-Grained Policy Optimizationcited by 23
  2. 2025Causal Tracing of Object Representations in Large Vision Language Models: Mechanistic Interpretability and Hallucination Mitigationcited by 22
  3. 2021An Information-Theoretic Perspective on Credit Assignment in Reinforcement Learningcited by 20
  4. 2025CLAIM: Mitigating Multilingual Object Hallucination in Large Vision-Language Models with Cross-Lingual Attention Interventioncited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 5.

5 papers.

Similar, not cited

Close in wording, no citation either way

  1. 2026PRPO: Perception-Reinforced Policy Optimization via Token-Level Dynamic Advantage Reshaping
  2. 2026Reinforcing Multimodal Reasoning via Token-Level Perception-Grounded Advantage Estimation
  3. 2026Rethinking Token-Level Policy Optimization for Multimodal Chain-of-Thought
  4. 2026Bridging Perception and Reasoning: Token Reweighting for RLVR in Multimodal LLMs
  5. 2026Visually-Guided Policy Optimization for Multimodal Reasoning
  6. 2026Thinking with Deltas: Incentivizing Reinforcement Learning via Differential Visual Reasoning Policy
  7. 2026PDCR: Perception-Decomposed Confidence Reward for Vision-Language Reasoning
  8. 2025More Thought, Less Accuracy? On the Dual Nature of Reasoning in Vision-Language Models