# Spotlight on Token Perception for Multimodal Reinforcement Learning

arXiv: https://arxiv.org/abs/2510.09285 · HTML: https://mlsanity.com/paper/2510.09285/

## Key facts

- Cited by 39 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal reasoning, one of 17 directions in LLM reasoning.
- Builds on 55 of its 69 references that are in the map.
- First posted on arXiv 10 Oct 2025.

## Placed in

- LLM reasoning / [Multimodal reasoning](https://mlsanity.com/lr/multimodal-reasoning/)
- Multimodal LLMs / vision-language models / [Embodied and 3D vision-language models](https://mlsanity.com/mllm/embodied-and-3d-vision-language-models/)
- Deep reinforcement learning methods / [RL training of language and agent models](https://mlsanity.com/deeprl/rl-training-of-language-and-agent-models/)

## Builds on (most cited first, 30 of 55)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Qwen2.5-VL Technical Report](https://mlsanity.com/paper/2502.13923/) (2025)
- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024)
- [DAPO: An Open-Source LLM Reinforcement Learning System at Scale](https://mlsanity.com/paper/2503.14476/) (2025)
- [HybridFlow: A Flexible and Efficient RLHF Framework](https://mlsanity.com/paper/2409.19256/) (2024)
- [Kimi k1.5: Scaling Reinforcement Learning with LLMs](https://mlsanity.com/paper/2501.12599/) (2025)
- [A-OKVQA: A Benchmark for Visual Question Answering using World Knowledge](https://mlsanity.com/paper/2206.01718/) (2022)
- [Vision-R1: Incentivizing Reasoning Capability in Multimodal Large Language Models](https://mlsanity.com/paper/2503.06749/) (2025)
- [MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?](https://mlsanity.com/paper/2403.14624/) (2024)
- [Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning](https://mlsanity.com/paper/2506.01939/) (2025)
- [Measuring Multimodal Mathematical Reasoning with MATH-Vision Dataset](https://mlsanity.com/paper/2402.14804/) (2024)
- [VLM-R1: A Stable and Generalizable R1-style Large Vision-Language Model](https://mlsanity.com/paper/2504.07615/) (2025)
- [MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmark](https://mlsanity.com/paper/2409.02813/) (2024)
- [DeepEyes: Incentivizing "Thinking with Images" via Reinforcement Learning](https://mlsanity.com/paper/2505.14362/) (2025)
- [Inter-GPS: Interpretable Geometry Problem Solving with Formal Language and Symbolic Reasoning](https://mlsanity.com/paper/2105.04165/) (2021)
- [R1-Onevision: Advancing Generalized Multimodal Reasoning through Cross-Modal Formalization](https://mlsanity.com/paper/2503.10615/) (2025)
- [Absolute Zero: Reinforced Self-play Reasoning with Zero Data](https://mlsanity.com/paper/2505.03335/) (2025)
- [VL-Rethinker: Incentivizing Self-Reflection of Vision-Language Models with Reinforcement Learning](https://mlsanity.com/paper/2504.08837/) (2025)
- [MM-Eureka: Exploring the Frontiers of Multimodal Reasoning with Rule-based Reinforcement Learning](https://mlsanity.com/paper/2503.07365/) (2025)
- [We-Math: Does Your Large Multimodal Model Achieve Human-like Mathematical Reasoning?](https://mlsanity.com/paper/2407.01284/) (2024)
- [Learning to Reason under Off-Policy Guidance](https://mlsanity.com/paper/2504.14945/) (2025)
- [LogicVista: Multimodal LLM Logical Reasoning Benchmark in Visual Contexts](https://mlsanity.com/paper/2407.04973/) (2024)
- [Enhancing the Reasoning Ability of Multimodal Large Language Models via Mixed Preference Optimization](https://mlsanity.com/paper/2411.10442/) (2024)
- [Thinking with Images for Multimodal Reasoning: Foundations, Methods, and Future Frontiers](https://mlsanity.com/paper/2506.23918/) (2025)
- [A Survey of Reinforcement Learning for Large Reasoning Models](https://mlsanity.com/paper/2509.08827/) (2025)
- [OpenThinkIMG: Learning to Think with Images via Visual Tool Reinforcement Learning](https://mlsanity.com/paper/2505.08617/) (2025)

## Built on by (most cited first, 30 of 39)

- [Reasoning Within the Mind: Dynamic Multimodal Interleaving in Latent Space](https://mlsanity.com/paper/2512.12623/) (2025)
- [Visual-Advantage On-Policy Distillation for Vision-Language Models](https://mlsanity.com/paper/2605.21924/) (2026)
- [DiffThinker: Towards Generative Multimodal Reasoning with Diffusion Models](https://mlsanity.com/paper/2512.24165/) (2025)
- [Visually-Guided Policy Optimization for Multimodal Reasoning](https://mlsanity.com/paper/2604.09349/) (2026)
- [Skill Self-Play: Pushing the Frontier of LLM Capability with Co-Evolving Skills](https://mlsanity.com/paper/2607.22529/) (2026)
- [Rethinking Token-Level Policy Optimization for Multimodal Chain-of-Thought](https://mlsanity.com/paper/2603.22847/) (2026)
- [Seeing with You: Perception-Reasoning Coevolution for Multimodal Reasoning](https://mlsanity.com/paper/2603.28618/) (2026)
- [Not All Tokens See Equally: Perception-Grounded Policy Optimization for Large Vision-Language Models](https://mlsanity.com/paper/2604.01840/) (2026)
- [Credit Where It is Due: Cross-Modality Connectivity Drives Precise Reinforcement Learning for MLLM Reasoning](https://mlsanity.com/paper/2602.11455/) (2026)
- [Self-Supervised Visual On-Policy Distillation](https://mlsanity.com/paper/2608.14144/) (2026)
- [CLIPO: Contrastive Learning in Policy Optimization Generalizes RLVR](https://mlsanity.com/paper/2603.10101/) (2026)
- [PRPO: Perception-Reinforced Policy Optimization via Token-Level Dynamic Advantage Reshaping](https://mlsanity.com/paper/2606.08708/) (2026)
- [CARE What Fails: Contrastive Anchored-REflection for Verifiable Multimodal Reasoning](https://mlsanity.com/paper/2512.19554/) (2025)
- [Persistent Visual Memory: Sustaining Perception for Deep Generation in LVLMs](https://mlsanity.com/paper/2605.00814/) (2026)
- [Attend to Evidence: Evidence-Anchored Spatial Attention Supervision for Multimodal RLVR](https://mlsanity.com/paper/2605.30912/) (2026)
- [PDCR: Perception-Decomposed Confidence Reward for Vision-Language Reasoning](https://mlsanity.com/paper/2605.13467/) (2026)
- [Escape the Language Prior: Mitigating Late-Stage Modality Collapse in Audio Reasoning via Modality-Aware Policy Optimization](https://mlsanity.com/paper/2605.27741/) (2026)
- [Evidence-RL: Towards Evidence-intensive Visual Reasoning](https://mlsanity.com/paper/2608.08021/) (2026)
- [StAR: Segment Anything Reasoner](https://mlsanity.com/paper/2603.14382/) (2026)
- [Bridging Visual Representation and Reinforcement Learning from Verifiable Rewards in Large Vision-Language Models](https://mlsanity.com/paper/2603.27375/) (2026)
- [Veritas++: Value-aware On-Policy Distillation for Perception-Enhanced AIGI Detection](https://mlsanity.com/paper/2607.27113/) (2026)
- [CPPO: Contrastive Perception Policy Optimization for VLM Agents](https://mlsanity.com/paper/2601.00501/) (2026)
- [Entropy Is Not Enough: Unlocking Effective Reinforcement Learning for Visual Reasoning via Vision-Anchored Token Selection](https://mlsanity.com/paper/2606.03937/) (2026)
- [Teaching the Way, Not the Answer: Privileged Tutoring Distillation for Multimodal Policy Optimization](https://mlsanity.com/paper/2606.07000/) (2026)
- [DyCo-RL: Dynamic Cross-Modal Coordination for Visual Reasoning](https://mlsanity.com/paper/2606.08035/) (2026)
- [Reflection Anchors for Propagation-Aware Visual Retention in Long-Chain Multimodal Reasoning](https://mlsanity.com/paper/2605.09614/) (2026)
- [Guidance Contrastive Token Credit Assignment for Discrete Policy Optimization](https://mlsanity.com/paper/2605.29198/) (2026)
- [SIVA-RL: Sensitivity-Invariance Visual Alignment for Multimodal Reinforcement Learning](https://mlsanity.com/paper/2607.13931/) (2026)
- [Correcting What You Cannot See: Credit Assignment for Perception Distillation in Multimodal Reasoners](https://mlsanity.com/paper/2607.28336/) (2026)
- [Balancing Efficiency and Efficacy: Training-Free Attention-Guided Switching Between Explicit and Latent Thoughts for MLLMs](https://mlsanity.com/paper/2608.03450/) (2026)

## Abstract (arXiv)

While Reinforcement Learning with Verifiable Rewards (RLVR) has advanced the reasoning capabilities of Large Vision-Language Models (LVLMs), most existing methods in multimodal reasoning neglect the critical role of visual perception within the RLVR optimization process. In this paper, we undertake a pioneering exploration of multimodal RLVR through the novel perspective of token perception, which measures the visual dependency of each generated token. With a granular analysis of Chain-of-Thought (CoT) processes, we uncover two key insights: first, token perception in a rollout trajectory is sparsely distributed, where only a small fraction of tokens have high visual dependency for visually-grounded reasoning; second, different trajectories exhibit significant divergence in their overall visual dependency. Based on these observations, we propose Visually-Perceptive Policy Optimization (VPPO), a novel policy gradient algorithm that explicitly leverages token perception to refine the learning signal. Specifically, VPPO achieves this through a dual mechanism: it reweights a trajectory's advantage by its overall visual dependency, and focuses policy updates exclusively on perceptually pivotal tokens. On a comprehensive suite of eight perception and reasoning benchmarks, VPPO demonstrates substantial gains over leading open-source RL-tuned models, with its effectiveness consistently validated across 7B and 32B model scales. Our findings not only establish a new token-level perceptual perspective for analyzing multimodal RLVR but also present a novel and effective optimization strategy to significantly enhance the multimodal reasoning capabilities of LVLMs.