Training Data Efficiency in Multimodal Process Reward Models
On the map: LLM reasoning → Multimodal reasoning
also in 1 more
Abstract (arXiv)
Multimodal Process Reward Models (MPRMs) are central to step-level supervision for visual reasoning in MLLMs. Training MPRMs typically requires large-scale Monte Carlo (MC)-annotated corpora, incurring substantial training cost. This paper studies the data efficiency for MPRM training. Our preliminary experiments reveal that MPRM training quickly saturates under random subsampling of the training data, indicating substantial redundancy within existing MC-annotated corpora. To explain this, we formalize a theoretical framework and reveal that informative gradient updates depend on two factors: label mixtures of positive/negative steps and label reliability (average MC scores of positive steps). Guided by these insights, we propose the Balanced-Information Score (BIS), which prioritizes both mixture and reliability based on existing MC signals at the rollout level, without incurring any additional cost. Across two backbones (InternVL2.5-8B and Qwen2.5-VL-7B) on VisualProcessBench, BIS-selected subsets consistently match and even surpass the full-data performance at small fractions. Notably, the BIS subset reaches full-data performance using only 10% of the training data, improving over random subsampling by a relative 4.1%.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 36 of its 89 references that are in the map. First posted on arXiv 4 Feb 2026.
Builds on
36 of 89 references.
Closest ancestors · 11
Same direction, newest first
- 2025TIM-PRM: Verifying multimodal reasoning with Tool-Integrated PRM28 Nov
- 2025Training Vision-Language Process Reward Models for Test-Time Scaling in Multimodal Reasoning: Key Insights and Lessons Learned27 Sep
- 2025GM-PRM: A Generative Multimodal Process Reward Model for Multimodal Mathematical Reasoning6 Aug
- 2025DreamPRM: Domain-Reweighted Process Reward Model for Multimodal Reasoning26 May
- 2025MM-PRM: Enhancing Multimodal Mathematical Reasoning with Scalable Step-Level Supervision19 May
- 2025Skywork-VL Reward: An Effective Reward Model for Multimodal Understanding and Reasoning12 May
- 2025ViLBench: A Suite for Vision-Language Process Reward Modeling26 Mar
- 2025VisualPRM: An Effective Process Reward Model for Multimodal Reasoning13 Mar
- 2024Diving into Self-Evolving Training for Multimodal Reasoning23 Dec
- 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling6 Dec
- 2024Enhancing the Reasoning Ability of Multimodal Large Language Models via Mixed Preference Optimization15 Nov
Other references · 25
- 2025Qwen2.5-VL Technical Reportcited by 5,418
- 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
- 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
- 2025Gemma 3 Technical Reportcited by 1,686
- 2025Process Reinforcement through Implicit Rewardscited by 478
- 2024Improve Mathematical Reasoning in Language Models by Automated Process Supervisioncited by 426
- 2024Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Expertscited by 369
- 2017FigureQA: An Annotated Figure Dataset for Visual Reasoningcited by 314
- 2025Process Reward Models That Thinkcited by 118
- 2023Let's reward step by step: Step-Level reward model as the Navigators for Reasoningcited by 96
- 2024OpenR: An Open Source Framework for Advanced Reasoning with Large Language Modelscited by 92
- 2025R1-Reward: Training Multimodal Reward Model Through Stable Reinforcement Learningcited by 65
13 more
- 2025SophiaVL-R1: Reinforcing MLLMs Reasoning with Thinking Rewardcited by 45
- 2025A Survey of Process Reward Models: From Outcome Signals to Process Supervisions for Large Language Modelscited by 37
- 2025StepWiser: Stepwise Generative Judges for Wiser Reasoningcited by 28
- 2025Retrieval-Augmented Process Reward Model for Generalizable Mathematical Reasoningcited by 23
- 2025AURORA:Automated Training Framework of Universal Process Reward Models via Ensemble Prompting and Reverse Verificationcited by 22
- 2025Athena: Enhancing Multimodal Reasoning with Data-efficient Process Reward Modelscited by 16
- 2025BaseReward: A Strong Baseline for Multimodal Reward Modelcited by 15
- 2025Efficient Process Reward Model Training via Active Learningcited by 14
- 2025An Efficient and Precise Training Data Construction Framework for Process-supervised Reward Model in Mathematical Reasoningcited by 12
- 2025PRM-BAS: Enhancing Multimodal Reasoning through PRM-guided Beam Annealing Searchcited by 12
- 2025VRPRM: Process Reward Modeling via Visual Reasoningcited by 11
- 2025Uncertainty-Based Methods for Automated Process Reward Data Construction and Output Aggregation in Mathematical Reasoningcited by 2
- 2025DreamPRM-1.5: Unlocking the Potential of Each Instance for Multimodal Process Reward Model Trainingcited by 1
Built on by
Citing papers per year: 2026: 1.
1 paper.
- 2026Process Rewards with Learned Reliabilitycited by 0
- 2026Process Rewards with Learned Reliability15 May 2026
Similar, not cited
Close in wording, no citation either way
- 2025MPBench: A Comprehensive Multimodal Reasoning Benchmark for Process Errors Identification
- 2025Benchmarking Multimodal CoT Reward Model Stepwise by Visual Program
- 2025Multimodal RewardBench: Holistic Evaluation of Reward Models for Vision Language Models
- 2025Multimodal RewardBench 2: Evaluating Omni Reward Models for Interleaved Text and Image
- 2025Evaluating MLLMs with Multimodal Multi-image Reasoning Benchmark
- 2026Entropy-Guided Data-Efficient Training for Multimodal Reasoning Reward Models
- 2025Unlocking Multimodal Mathematical Reasoning via Process Reward Model
- 2026MMVistaReason: Toward Open-Data and Post-Training Recipes for Multimodal Reasoning