mlsanityData to 7 Oct 2026

Transferring Textual Preferences to Vision-Language Understanding through Model Merging

arXiv 2502.13487 ↗First posted 19 Feb 2025cs.CL cs.AI cs.CVcited by 3

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

also in 1 more
Abstract (arXiv)

Large vision-language models (LVLMs) perform outstandingly across various multimodal tasks. However, their ability to evaluate generated content remains limited, and training vision-language reward models (VLRMs) with preference data is computationally expensive. This paper explores a training-free alternative by merging text-based reward models (RMs) with LVLMs to create VLRMs. Our approach shows that integrating these models leads to improved performance over LVLMs' scoring and text-based RMs, offering an efficient method for incorporating textual preferences into LVLMs.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 32 of its 49 references that are in the map. First posted on arXiv 19 Feb 2025.

Builds on

32 of 49 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024Synth-Align: Improving Trustworthiness in Vision-Language Model with Synthetic Preference Data Alignment23 Dec
  2. 2024MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmark7 Feb
  3. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captions21 Nov
  4. 2023Aligning Large Multimodal Models with Factually Augmented RLHF25 Sep

Other references · 28

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2024The Llama 3 Herd of Modelscited by 14,192
  4. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  5. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  6. 2020Learning to summarize from human feedbackcited by 2,562
  7. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  8. 2019Towards VQA Models That Can Readcited by 1,696
  9. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
  10. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 738
  11. 2024DeepSeek-VL: Towards Real-World Vision-Language Understandingcited by 718
  12. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignmentcited by 712
16 more
  1. 2023Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunchcited by 679
  2. 2024WildChat: 1M ChatGPT Interaction Logs in the Wildcited by 598
  3. 2024MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmarkcited by 494
  4. 2024Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Modelscited by 431
  5. 2021Understanding Dataset Difficulty with V-Usable Informationcited by 288
  6. 2024Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunitiescited by 247
  7. 2024HelpSteer2: Open-source dataset for training top-performing reward modelscited by 232
  8. 2024Aligning Modalities in Vision Large Language Models via Preference Fine-tuningcited by 197
  9. 2024Arcee's MergeKit: A Toolkit for Merging Large Language Modelscited by 189
  10. 2024WARM: On the Benefits of Weight Averaged Reward Modelscited by 145
  11. 2024Prometheus-Vision: Vision-Language Model as a Judge for Fine-Grained Evaluationcited by 123
  12. 2024Unpacking DPO and PPO: Disentangling Best Practices for Learning from Preference Feedbackcited by 107
  13. 2024VL-RewardBench: A Challenging Benchmark for Vision-Language Generative Reward Modelscited by 85
  14. 2024Detecting and Mitigating Hallucination in Large Vision Language Models via Fine-Grained AI Feedbackcited by 80
  15. 2024VLFeedback: A Large-Scale AI Feedback Dataset for Large Vision-Language Models Alignmentcited by 56
  16. 2024DogeRM: Equipping Reward Models with Domain Knowledge through Model Mergingcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 1.

3 papers.

  1. 2025VisCodex: Unified Multimodal Code Generation via Merging Vision and Coding Modelscited by 13
  2. 2025Model Merging to Maintain Language-Only Performance in Developmentally Plausible Multimodal Modelscited by 1
  3. 2026Investigating Cross-Modal Skill Injection: Scenarios, Methods, and Hyperparameterscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025From Captions to Rewards (CAREVL): Leveraging Large Language Model Experts for Enhanced Reward Modeling in Large Vision-Language Models
  2. 2025Multimodal RewardBench: Holistic Evaluation of Reward Models for Vision Language Models
  3. 2025InternLM-XComposer2.5-Reward: A Simple Yet Effective Multi-Modal Reward Model
  4. 2024Calibrated Self-Rewarding Vision Language Models
  5. 2026Lost in Translation: Do LVLM Judges Generalize Across Languages?
  6. 2026Learning What Matters: Dynamic Dimension Selection and Aggregation for Interpretable Vision-Language Reward Modeling
  7. 2025VLRMBench: A Comprehensive and Challenging Benchmark for Vision-Language Reward Models
  8. 2025ViLBench: A Suite for Vision-Language Process Reward Modeling