Residual Visual Credit Optimization: Conserved Evidence Routing for Multimodal Reinforcement Learning
On the map: LLM reasoning → Reinforcement learning for reasoning
also in 1 more
Abstract (arXiv)
Reinforcement learning with verifiable rewards scales multimodal reasoning, but an outcome reward says how much a trajectory is worth, not how that value should be spread over the decisions that produced it. We introduce Residual Visual Credit Optimization (RVCO), which treats token credit as a conserved routing problem. A controlled visual intervention yields a per-token evidence response; robust within-trajectory coordinates remove incidental scale; and a budgeted entropic router distributes a fixed amount of sequence utility according to perceptual dependence. A residual support path guarantees positive credit at every valid position, and an analytic correction restores the prescribed credit mass exactly. The resulting field is selective, bounded, full-support, and invariant to response-local score shifts, and recovers hard token selection as a limiting case. Across four model families and seven reasoning benchmarks, RVCO improves accuracy over strong RLVR baselines while maintaining late-stage optimization stability, corruption robustness, and competitive training cost. Rewards, rollouts, and the group-relative advantage estimator are unchanged; only the geometry of token-level credit differs.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 44 of its 51 references that are in the map. First posted on arXiv 4 Oct 2026.
Builds on
44 of 51 references.
Closest ancestors · 4
Same direction, newest first
- 2025Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMs17 Jun
- 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
- 2023Let's Verify Step by Step31 May
- 2017Proximal Policy Optimization Algorithms20 Jul
Other references · 40
- 2017Attention Is All You Needcited by 60,361
- 2024The Llama 3 Herd of Modelscited by 14,715
- 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
- 2025Qwen2.5-VL Technical Reportcited by 5,587
- 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
- 2025Qwen3-VL Technical Reportcited by 2,793
- 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
- 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,224
- 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,834
- 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,479
- 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,317
- 2021DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsificationcited by 864
28 more
- 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 710
- 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decodingcited by 688
- 2025Vision-R1: Incentivizing Reasoning Capability in Multimodal Large Language Modelscited by 665
- 2024MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?cited by 586
- 2023HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Modelscited by 583
- 2025Visual-RFT: Visual Reinforcement Fine-Tuningcited by 513
- 2024MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmarkcited by 512
- 2021InfographicVQAcited by 437
- 2021Inter-GPS: Interpretable Geometry Problem Solving with Formal Language and Symbolic Reasoningcited by 383
- 2025VL-Rethinker: Incentivizing Self-Reflection of Vision-Language Models with Reinforcement Learningcited by 323
- 2024LogicVista: Multimodal LLM Logical Reasoning Benchmark in Visual Contextscited by 245
- 2024Multi-Modal Hallucination Control by Visual Information Groundingcited by 208
- 2024DynaMath: A Dynamic Visual Benchmark for Evaluating Mathematical Reasoning Robustness of Vision Language Modelscited by 146
- 2019Learning with Fenchel-Young Lossescited by 119
- 2025Perception-Aware Policy Optimization for Multimodal Reasoningcited by 99
- 2025SimpleVQA: Multimodal Factuality Evaluation for Multimodal Large Language Modelscited by 66
- 2023UPop: Unified and Progressive Pruning for Compressing Vision-Language Transformerscited by 54
- 2025Spotlight on Token Perception for Multimodal Reinforcement Learningcited by 39
- 2025Mitigating Visual Forgetting via Take-along Visual Conditioning for Multi-modal Long CoT Reasoningcited by 32
- 2019The Limited Multi-Label Projection Layercited by 22
- 2025VisNumBench: Evaluating Number Sense of Multimodal Large Language Modelscited by 14
- 2026Visually-Guided Policy Optimization for Multimodal Reasoningcited by 11
- 2026Rethinking Token-Level Policy Optimization for Multimodal Chain-of-Thoughtcited by 9
- 2026VL-Calibration: Decoupled Confidence Calibration for Large Vision-Language Models Reasoningcited by 8
- 2026Not All Tokens See Equally: Perception-Grounded Policy Optimization for Large Vision-Language Modelscited by 7
- 2026PRPO: Perception-Reinforced Policy Optimization via Token-Level Dynamic Advantage Reshapingcited by 5
- 2026Learning Self-Correction in Vision-Language Models via Rollout Augmentationcited by 4
- 2026Improving Vision-language Models with Perception-centric Process Reward Modelscited by 4
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026Visual sensitivity is not claim retractability: persistence-aware credit assignment for multimodal reinforcement learning
- 2026Entropy Is Not Enough: Unlocking Effective Reinforcement Learning for Visual Reasoning via Vision-Anchored Token Selection
- 2026Structured Role-Aware Policy Optimization for Multimodal Reasoning
- 2026Beyond Entropy: Self-Diagnostic Multi-Role Token Optimization for Video Reasoning
- 2026SIVA-RL: Sensitivity-Invariance Visual Alignment for Multimodal Reinforcement Learning
- 2026Credit Where It is Due: Cross-Modality Connectivity Drives Precise Reinforcement Learning for MLLM Reasoning
- 2026IVR-R1: Refining Trajectories through Iterative Visual-Grounded Reasoning in Reinforcement Learning
- 2026Bridging Perception and Reasoning: Token Reweighting for RLVR in Multimodal LLMs