mlsanityData to 7 Oct 2026

Difference Feedback: Generating Multimodal Process-Level Supervision for VLM Reinforcement Learning

arXiv 2603.27482 ↗First posted 29 Mar 2026cs.CV cs.AIcited by 0

On the map: LLM reasoning → Multimodal reasoning

Abstract (arXiv)

Vision--language models (VLMs) are increasingly aligned via Group Relative Policy Optimization (GRPO)-style training. However, relying solely on terminal outcome rewards yields sparse credit assignment in multi-step reasoning, weakening the linkage between visual evidence and intermediate steps and often causing unstable optimization and visual hallucinations. We propose Differential Feedback, which automatically constructs token/step-level supervision masks by repairing erroneous reasoning trajectories, explicitly marking the key positions that require correction. Without costly large-scale step-by-step human annotations, our method enables process-level visual alignment and can be seamlessly integrated into existing GRPO-like frameworks. Experiments on multimodal reasoning benchmarks including MMMStar and MathVista show an average 3% improvement under matched compute budgets. Our approach offers an effective, low-cost solution for accurate vision--reasoning process alignment.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 24 of its 25 references that are in the map. First posted on arXiv 29 Mar 2026.

Builds on

24 of 25 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency25 Aug
  2. 2025Vision-G1: Towards General Vision Language Reasoning with Multi-Domain Data Curation18 Aug
  3. 2025R1-ShareVL: Incentivizing Reasoning Capability of Multimodal Large Language Models via Share-GRPO22 May
  4. 2025SoTA with Less: MCTS-Guided Sample Selection for Data-Efficient Visual Reasoning Self-Improvement10 Apr
  5. 2025Vision-R1: Incentivizing Reasoning Capability in Multimodal Large Language Models9 Mar

Other references · 19

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  4. 2025Qwen2.5-VL Technical Reportcited by 5,418
  5. 2023Let's Verify Step by Stepcited by 3,598
  6. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  7. 2025Group Sequence Policy Optimizationcited by 788
  8. 2025Process Reinforcement through Implicit Rewardscited by 478
  9. 2024Training Language Models to Self-Correct via Reinforcement Learningcited by 405
  10. 2024Free Process Rewards without Process Labelscited by 162
  11. 2024DPO Meets PPO: Reinforced Token Optimization for RLHFcited by 126
  12. 2020Explaining NLP Models via Minimal Contrastive Editing (MiCE)cited by 105
7 more
  1. 2025Soft Adaptive Policy Optimizationcited by 89
  2. 2024Dense Reward for Free in Reinforcement Learning from Human Feedbackcited by 77
  3. 2024Improving Large Language Models via Fine-grained Reinforcement Learning with Minimum Editing Constraintcited by 39
  4. 2025Discriminative Policy Optimization for Token-Level Reward Modelscited by 15
  5. 2024T-REG: Preference Optimization with Token-Level Reward Regularizationcited by 13
  6. 2025SCAR: Shapley Credit Assignment for More Efficient RLHFcited by 13
  7. 2024Inverse-Q*: Token Level Reinforcement Learning for Aligning Large Language Models Without Preference Datacited by 12

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026V-tableR1: Process-Supervised Multimodal Table Reasoning with Critic-Guided Policy Optimization
  2. 2026SIVA-RL: Sensitivity-Invariance Visual Alignment for Multimodal Reinforcement Learning
  3. 2026Anchoring What Matters: A Dual-Level Learning Framework for Visually-Grounded Multimodal Reasoning
  4. 2026Visually-Guided Policy Optimization for Multimodal Reasoning
  5. 2025PROPA: Toward Process-level Optimization in Visual Reasoning via Reinforcement Learning
  6. 2026IVR-R1: Refining Trajectories through Iterative Visual-Grounded Reasoning in Reinforcement Learning
  7. 2026PaLMR: Towards Faithful Visual Reasoning via Multimodal Process Alignment
  8. 2026Structured Role-Aware Policy Optimization for Multimodal Reasoning