mlsanityData to 14 Oct 2026

Unsupervised Multimodal Reward Modeling

arXiv 2610.12707 ↗First posted 8 Oct 2026cs.CL cs.AI cs.CVcited by 0

On the map: LLM alignment / RLHF / preference optimisation → Diffusion and multimodal model alignment

Abstract (arXiv)

Vision-language reward models are trained on annotated preference pairs, which are expensive to collect, can disagree, and are ranked by benchmark accuracy. We build one with no human preference label: a linear head on an intermediate layer of a frozen Qwen3-VL backbone, trained to score a web caption above a perturbed version of the same caption; labeled data enters only to select the layer. The layer is the decisive choice, worth 5.8 points over the last layer, and one recipe transferred without re-tuning beats or at least matches GPT-4o on three benchmarks at 32B, 6.6 points ahead on VL-RewardBench, from a few thousand parameters at about a thousandth of fine-tuning compute. At an equal pair budget, perturbations teach a frozen head 3 to 5 points more than curated RLAIF-V pairs. Because the backbone is frozen, the objective is convex, and the perturbation sampler specifies the reward, which the benchmark does not identify: two heads trained on disjoint halves of the same perturbations tie on the benchmark and differ by 6.4 points in hallucination judging. Retraining the head on the policy's own verifier-labeled candidates cuts caption hallucination by 5.1 CHAIR points and lifts image-generation pass rate by 5.4, while scoring lower on every reward benchmark. Together, an intermediate-layer probe, free same-image perturbations, and a verifier-driven bootstrap turn give a recipe for multimodal reward modeling that needs no annotator, costs a fraction of fine-tuning, and is specified by the comparisons it is trained on rather than by a benchmark rank.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Diffusion and multimodal model alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 65 of its 73 references that are in the map. First posted on arXiv 8 Oct 2026.

Builds on

65 of 73 references.

Closest ancestors · 14

Same direction, newest first

  1. 2025Multimodal RewardBench 2: Evaluating Omni Reward Models for Interleaved Text and Image18 Dec
  2. 2025Multimodal LLMs as Customized Reward Models for Text-to-Image Generation28 Jul
  3. 2025Unified Reward Model for Multimodal Understanding and Generation7 Mar
  4. 2025Multimodal RewardBench: Holistic Evaluation of Reward Models for Vision Language Models20 Feb
  5. 2025MM-RLHF: The Next Step Forward in Multimodal LLM Alignment14 Feb
  6. 2025InternLM-XComposer2.5-Reward: A Simple Yet Effective Multi-Modal Reward Model21 Jan
  7. 2024VL-RewardBench: A Challenging Benchmark for Vision-Language Generative Reward Models26 Nov
  8. 2024VLFeedback: A Large-Scale AI Feedback Dataset for Large Vision-Language Models Alignment12 Oct
  9. 2024RLAIF-V: Open-Source AI Feedback Leads to Super GPT-4V Trustworthiness27 May
  10. 2024Aligning Modalities in Vision Large Language Models via Preference Fine-tuning18 Feb
  11. 2023RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedback1 Dec
  12. 2023Beyond Hallucinations: Enhancing LVLMs through Hallucination-Aware Direct Preference Optimization28 Nov
2 more
  1. 2023Diffusion Model Alignment Using Direct Preference Optimization21 Nov
  2. 2023Aligning Large Multimodal Models with Factually Augmented RLHF25 Sep

Other references · 51

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  7. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  8. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  9. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  10. 2024GPT-4o System Cardcited by 4,588
  11. 2017Deep reinforcement learning from human preferencescited by 3,912
  12. 2023Let's Verify Step by Stepcited by 3,789
38 more
  1. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  2. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,877
  3. 2025Qwen3-VL Technical Reportcited by 2,793
  4. 2020Learning to summarize from human feedbackcited by 2,588
  5. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
  6. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,729
  7. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,614
  8. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,405
  9. 2016Understanding intermediate layers using linear classifier probescited by 1,353
  10. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,095
  11. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,059
  12. 2025Qwen-Image Technical Reportcited by 1,044
  13. 2023GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignmentcited by 972
  14. 2022STaR: Bootstrapping Reasoning With Reasoningcited by 940
  15. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 939
  16. 2022Solving math word problems with process- and outcome-based feedbackcited by 901
  17. 2022Discovering Latent Knowledge in Language Models Without Supervisioncited by 767
  18. 2018Object Hallucination in Image Captioningcited by 669
  19. 2023PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallelcited by 660
  20. 2020Contrastive Learning with Hard Negative Samplescited by 628
  21. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackcited by 625
  22. 2022When and why vision-language models behave like bags-of-words, and what to do about it?cited by 610
  23. 2024RewardBench: Evaluating Reward Models for Language Modelingcited by 499
  24. 2020Hard Negative Mixing for Contrastive Learningcited by 402
  25. 2023Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Modelscited by 349
  26. 2023Scaling Open-Vocabulary Object Detectioncited by 343
  27. 2025Layer by Layer: Uncovering Hidden Representations in Language Modelscited by 311
  28. 2023A Long Way to Go: Investigating Length Correlations in RLHFcited by 291
  29. 2017Thinking Fast and Slow with Deep Learning and Tree Searchcited by 252
  30. 2023SugarCrepe: Fixing Hackable Benchmarks for Vision-Language Compositionalitycited by 243
  31. 2023Reward Model Ensembles Help Mitigate Overoptimizationcited by 231
  32. 2025RewardBench 2: Advancing Reward Model Evaluationcited by 150
  33. 2024LLaVA-Critic: Learning to Evaluate Multimodal Modelscited by 134
  34. 2024Theoretical guarantees on the best-of-n alignment policycited by 124
  35. 2018Noise Contrastive Estimation and Negative Sampling for Conditional Models: Consistency and Statistical Efficiencycited by 105
  36. 2024How to Evaluate Reward Models for RLHFcited by 88
  37. 2024Implicit Bias of AdamW: ℓ_∞ Norm Constrained Optimizationcited by 58
  38. 2024Rethinking Reward Model Evaluation: Are We Barking up the Wrong Tree?cited by 31

50 most cited of 51 All 51 (JSON)

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025From Captions to Rewards (CAREVL): Leveraging Large Language Model Experts for Enhanced Reward Modeling in Large Vision-Language Models
  2. 2026You Only Judge Once: Multi-response Reward Modeling in a Single Forward Pass
  3. 2026DeltaRubric: Generative Multimodal Reward Modeling via Joint Planning and Verification
  4. 2025Revisiting the Learning Objectives of Vision-Language Reward Models
  5. 2025Transferring Textual Preferences to Vision-Language Understanding through Model Merging
  6. 2026Joint Reward Modeling: Internalizing Chain-of-Thought for Efficient Visual Reward Models
  7. 2025ViLBench: A Suite for Vision-Language Process Reward Modeling
  8. 2026PromptEcho: Annotation-Free Reward from Vision-Language Models for Text-to-Image Reinforcement Learning