mlsanityData to 7 Oct 2026

InternLM-XComposer2.5-Reward: A Simple Yet Effective Multi-Modal Reward Model

arXiv 2501.12368 ↗First posted 21 Jan 2025cs.CV cs.CLcited by 73

On the map: LLM reasoning → Multimodal reasoning

also in 2 more
Abstract (arXiv)

Despite the promising performance of Large Vision Language Models (LVLMs) in visual understanding, they occasionally generate incorrect outputs. While reward models (RMs) with reinforcement learning or test-time scaling offer the potential for improving generation quality, a critical gap remains: publicly available multi-modal RMs for LVLMs are scarce, and the implementation details of proprietary models are often unclear. We bridge this gap with InternLM-XComposer2.5-Reward (IXC-2.5-Reward), a simple yet effective multi-modal reward model that aligns LVLMs with human preferences. To ensure the robustness and versatility of IXC-2.5-Reward, we set up a high-quality multi-modal preference corpus spanning text, image, and video inputs across diverse domains, such as instruction following, general understanding, text-rich documents, mathematical reasoning, and video understanding. IXC-2.5-Reward achieves excellent results on the latest multi-modal reward model benchmark and shows competitive performance on text-only reward model benchmarks. We further demonstrate three key applications of IXC-2.5-Reward: (1) Providing a supervisory signal for RL training. We integrate IXC-2.5-Reward with Proximal Policy Optimization (PPO) yields IXC-2.5-Chat, which shows consistent improvements in instruction following and multi-modal open-ended dialogue; (2) Selecting the best response from candidate responses for test-time scaling; and (3) Filtering outlier or noisy samples from existing image and video instruction tuning training data. To ensure reproducibility and facilitate further research, we have open-sourced all model weights and training recipes at https://github.com/InternLM/InternLM-XComposer/tree/main/InternLM-XComposer-2.5-Reward

Key facts. Cited by 73 papers in the map (data through 7 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 98 of its 123 references that are in the map. First posted on arXiv 21 Jan 2025.

Builds on

98 of 123 references.

Closest ancestors · 10

Same direction, newest first

  1. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling6 Dec
  2. 2024Scaling Inference-Time Search with Vision Value Model for Improved Visual Comprehension4 Dec
  3. 2024Critic-V: VLM Critics Help Catch VLM Errors in Multimodal Reasoning27 Nov
  4. 2024Measuring Multimodal Mathematical Reasoning with MATH-Vision Dataset22 Feb
  5. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts3 Oct
  6. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul
  7. 2022Super-CLEVR: A Virtual Benchmark to Diagnose Domain Robustness in Visual Reasoning1 Dec
  8. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering20 Sep
  9. 2022CLEVR-Math: A Dataset for Compositional Language, Visual and Mathematical Reasoning10 Aug
  10. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning19 Mar

Other references · 88

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2017Proximal Policy Optimization Algorithmscited by 14,441
  4. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  5. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  6. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  7. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  8. 2024GPT-4o System Cardcited by 4,467
  9. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  10. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  11. 2023Let's Verify Step by Stepcited by 3,598
  12. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
38 more
  1. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  2. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  3. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,928
  4. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  5. 2019Towards VQA Models That Can Readcited by 1,696
  6. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
  7. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,151
  8. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,061
  9. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
  10. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 988
  11. 2022STaR: Bootstrapping Reasoning With Reasoningcited by 895
  12. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 893
  13. 2022Solving math word problems with process- and outcome-based feedbackcited by 871
  14. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 865
  15. 2016A Diagram Is Worth A Dozen Imagescited by 793
  16. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 703
  17. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 702
  18. 2024Self-Rewarding Language Modelscited by 654
  19. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 630
  20. 2024MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?cited by 573
  21. 2023Reinforced Self-Training (ReST) for Language Modelingcited by 486
  22. 2024RewardBench: Evaluating Reward Models for Language Modelingcited by 480
  23. 2023OCRBench: On the Hidden Mystery of OCR in Large Multimodal Modelscited by 466
  24. 2023RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedbackcited by 404
  25. 2023UltraFeedback: Boosting Language Models with Scaled AI Feedbackcited by 394
  26. 2024InternLM2 Technical Reportcited by 391
  27. 2024ShareGPT4Video: Improving Video Understanding and Generation with Better Captionscited by 386
  28. 2024Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Expertscited by 369
  29. 2022Dynamic Prompt Learning via Policy Gradient for Semi-structured Mathematical Reasoningcited by 347
  30. 2018DVQA: Understanding Data Visualizations via Question Answeringcited by 338
  31. 2023PMC-VQA: Visual Instruction Tuning for Medical Visual Question Answeringcited by 333
  32. 2024Skywork-Reward: Bag of Tricks for Reward Modeling in LLMscited by 322
  33. 2024Long-CLIP: Unlocking the Long-Text Capability of CLIPcited by 298
  34. 2021Understanding Dataset Difficulty with V-Usable Informationcited by 288
  35. 2024ProcessBench: Identifying Process Errors in Mathematical Reasoningcited by 277
  36. 2024Rewarding Progress: Scaling Automated Process Verifiers for LLM Reasoningcited by 267
  37. 2021IconQA: A New Benchmark for Abstract Diagram Understanding and Visual Language Reasoningcited by 262
  38. 2023Camels in a Changing Climate: Enhancing LM Adaptation with Tulu 2cited by 227

50 most cited of 88 All 88 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 49; 2026: 24.

73 papers.

Sort:

By area:

  1. 2025VLM-R1: A Stable and Generalizable R1-style Large Vision-Language Modelcited by 515
  2. 2025Visual-RFT: Visual Reinforcement Fine-Tuningcited by 501
  3. 2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 381
  4. 2025SFT or RL? An Early Investigation into Training R1-Like Reasoning Large Vision-Language Modelscited by 209
  5. 2025Skywork-Reward-V2: Scaling Preference Data Curation via Human-AI Synergycited by 204
  6. 2025Unified Reward Model for Multimodal Understanding and Generationcited by 194
  7. 2025Unified Multimodal Chain-of-Thought Reward Model through Reinforcement Fine-Tuningcited by 86
  8. 2025R1-Reward: Training Multimodal Reward Model Through Stable Reinforcement Learningcited by 65
  9. 2025VideoRoPE: What Makes for Good Video Rotary Position Embedding?cited by 63
  10. 2025SIM-CoT: Supervised Implicit Chain-of-Thoughtcited by 63
  11. 2025Visual Agentic Reinforcement Fine-Tuningcited by 53
  12. 2025MathCoder-VL: Bridging Vision and Code for Enhanced Multimodal Mathematical Reasoningcited by 45
38 more
  1. 2025SophiaVL-R1: Reinforcing MLLMs Reasoning with Thinking Rewardcited by 45
  2. 2025MM-IFEngine: Towards Multimodal Instruction Followingcited by 41
  3. 2025LLaVA-Critic-R1: Your Critic Model is Secretly a Strong Policy Modelcited by 34
  4. 2025UI-Genie: A Self-Improving Approach for Iteratively Boosting MLLM-based Mobile GUI Agentscited by 33
  5. 2025Skywork-VL Reward: An Effective Reward Model for Multimodal Understanding and Reasoningcited by 30
  6. 2025Video-RTS: Rethinking Reinforcement Learning and Test-Time Scaling for Efficient and Enhanced Video Reasoningcited by 29
  7. 2025MM-PRM: Enhancing Multimodal Mathematical Reasoning with Scalable Step-Level Supervisioncited by 28
  8. 2025ManipLVM-R1: Reinforcement Learning for Reasoning in Embodied Manipulation with Large Vision-Language Modelscited by 26
  9. 2025Why Reasoning Matters? A Survey of Advancements in Multimodal Reasoning (v1)cited by 23
  10. 2025Evaluating MLLMs with Multimodal Multi-image Reasoning Benchmarkcited by 20
  11. 2025ViLBench: A Suite for Vision-Language Process Reward Modelingcited by 16
  12. 2025BaseReward: A Strong Baseline for Multimodal Reward Modelcited by 15
  13. 2025Multi-Crit: Benchmarking Multimodal Judges on Pluralistic Criteria-Followingcited by 14
  14. 2025Generative RLHF-V: Learning Principles from Multi-modal Human Preferencecited by 13
  15. 2025ARM-Thinker: Reinforcing Multimodal Generative Reward Models with Agentic Tool Use and Visual Reasoningcited by 13
  16. 2025AdsQA: Towards Advertisement Video Understandingcited by 12
  17. 2025Investigating Inference-time Scaling for Chain of Multi-modal Thought: A Preliminary Studycited by 11
  18. 2025EarthGPT-X: A Spatial MLLM for Multi-level Multi-Source Remote Sensing Imagery Understanding with Visual Promptingcited by 11
  19. 2025DocThinker: Explainable Multimodal Large Language Models with Rule-based Reinforcement Learning for Document Understandingcited by 11
  20. 2025VR-Thinker: Boosting Video Reward Models through Thinking-with-Image Reasoningcited by 11
  21. 2025Omni-Reward: Towards Generalist Omni-Modal Reward Modeling with Free-Form Preferencescited by 11
  22. 2026Unified Personalized Reward Model for Vision Generationcited by 11
  23. 2025From Captions to Rewards (CAREVL): Leveraging Large Language Model Experts for Enhanced Reward Modeling in Large Vision-Language Modelscited by 10
  24. 2025Benchmarking Multimodal CoT Reward Model Stepwise by Visual Programcited by 10
  25. 2025Boosting Virtual Agent Learning and Reasoning: A Step-Wise, Multi-Dimensional, and Generalist Reward Model with Benchmarkcited by 9
  26. 2025SPARK: Synergistic Policy And Reward Co-Evolving Frameworkcited by 9
  27. 2026PhyCritic: Multimodal Critic Models for Physical AIcited by 9
  28. 2025TIM-PRM: Verifying multimodal reasoning with Tool-Integrated PRMcited by 8
  29. 2026MMR-Life: Piecing Together Real-life Scenes for Multimodal Multi-image Reasoningcited by 8
  30. 2025MR. Judge: Multimodal Reasoner as a Judgecited by 7
  31. 2025Cooper: Co-Optimizing Policy and Reward Models in Reinforcement Learning for Large Language Modelscited by 5
  32. 2025VideoRewardBench: Comprehensive Evaluation of Multimodal Reward Models for Video Understandingcited by 5
  33. 2026Reward Modeling for Reinforcement Learning-Based LLM Reasoning: Design, Challenges, and Evaluationcited by 5
  34. 2026MSRL: Scaling Generative Multimodal Reward Modeling via Multi-Stage Reinforcement Learningcited by 5
  35. 2026Visual Preference Optimization with Rubric Rewardscited by 5
  36. 2025Multidimensional Rubric-oriented Reward Model Learning via Geometric Projection Reference Constraintscited by 4
  37. 2025Sightation Counts: Leveraging Sighted User Feedback in Building a BLV-aligned Dataset of Diagram Descriptionscited by 3
  38. 2025DSBench: A Comprehensive Benchmark for Evaluating External and In-Cabin Riskscited by 3

50 most cited of 73 All 73 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Multimodal RewardBench: Holistic Evaluation of Reward Models for Vision Language Models
  2. 2025Revisiting the Learning Objectives of Vision-Language Reward Models
  3. 2025Transferring Textual Preferences to Vision-Language Understanding through Model Merging
  4. 2025VLRMBench: A Comprehensive and Challenging Benchmark for Vision-Language Reward Models
  5. 2026Joint Reward Modeling: Internalizing Chain-of-Thought for Efficient Visual Reward Models
  6. 2025Vision-R1: Evolving Human-Free Alignment in Large Vision-Language Models via Vision-Guided Reinforcement Learning
  7. 2025VL-GenRM: Enhancing Vision-Language Verification via Vision Experts and Iterative Training
  8. 2024Calibrated Self-Rewarding Vision Language Models