mlsanityData to 7 Oct 2026

Omni-Reward: Towards Generalist Omni-Modal Reward Modeling with Free-Form Preferences

arXiv 2510.23451 ↗First posted 27 Oct 2025cs.CL cs.AI cs.CVcited by 11

On the map: LLM alignment / RLHF / preference optimisation → Diffusion and multimodal model alignment

Abstract (arXiv)

Reward models (RMs) play a critical role in aligning AI behaviors with human preferences, yet they face two fundamental challenges: (1) Modality Imbalance, where most RMs are mainly focused on text and image modalities, offering limited support for video, audio, and other modalities; and (2) Preference Rigidity, where training on fixed binary preference pairs fails to capture the complexity and diversity of personalized preferences. To address the above challenges, we propose Omni-Reward, a step toward generalist omni-modal reward modeling with support for free-form preferences, consisting of: (1) Evaluation: We introduce Omni-RewardBench, the first omni-modal RM benchmark with free-form preferences, covering nine tasks across five modalities including text, image, video, audio, and 3D; (2) Data: We construct Omni-RewardData, a multimodal preference dataset comprising 248K general preference pairs and 69K instruction-tuning pairs for training generalist omni-modal RMs; (3) Model: We propose Omni-RewardModel, which includes both discriminative and generative RMs, and achieves strong performance on Omni-RewardBench as well as other widely used reward modeling benchmarks.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Placed in Diffusion and multimodal model alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 72 of its 80 references that are in the map. First posted on arXiv 27 Oct 2025.

Builds on

72 of 80 references.

Closest ancestors · 23

Same direction, newest first

  1. 2025Aligning Multimodal LLM with Human Preference: A Survey18 Mar
  2. 2025VLRMBench: A Comprehensive and Challenging Benchmark for Vision-Language Reward Models10 Mar
  3. 2025Unified Reward Model for Multimodal Understanding and Generation7 Mar
  4. 2025OmniAlign-V: Towards Enhanced Alignment of MLLMs with Human Preference25 Feb
  5. 2025Multimodal RewardBench: Holistic Evaluation of Reward Models for Vision Language Models20 Feb
  6. 2025MM-RLHF: The Next Step Forward in Multimodal LLM Alignment14 Feb
  7. 2025Improving Video Generation with Human Feedback23 Jan
  8. 2025InternLM-XComposer2.5-Reward: A Simple Yet Effective Multi-Modal Reward Model21 Jan
  9. 2024VisionReward: Fine-Grained Multi-Dimensional Human Preference Learning for Image and Video Generation30 Dec
  10. 2024VideoDPO: Omni-Preference Alignment for Video Diffusion Generation18 Dec
  11. 2024LiFT: Leveraging Human Feedback for Text-to-Video Model Alignment6 Dec
  12. 2024Improving Dynamic Object Interactions in Text-to-Video Generation with AI Feedback3 Dec
11 more
  1. 2024VL-RewardBench: A Challenging Benchmark for Vision-Language Generative Reward Models26 Nov
  2. 2024Enhancing the Reasoning Ability of Multimodal Large Language Models via Mixed Preference Optimization15 Nov
  3. 2024MIA-DPO: Multi-Image Augmented Direct Preference Optimization For Large Vision-Language Models23 Oct
  4. 2024VLFeedback: A Large-Scale AI Feedback Dataset for Large Vision-Language Models Alignment12 Oct
  5. 2024MJ-Bench: Is Your Multimodal Reward Model Really a Good Judge for Text-to-Image Generation?5 Jul
  6. 2024Direct Preference Optimization of Video Large Multimodal Models from Language Model Reward1 Apr
  7. 2023Rich Human Feedback for Text-to-Image Generation15 Dec
  8. 2023RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedback1 Dec
  9. 2023Aligning Large Multimodal Models with Factually Augmented RLHF25 Sep
  10. 2023Human Preference Score: Better Aligning Text-to-Image Models with Human Preference25 Mar
  11. 2023Aligning Text-to-Image Models using Human Feedback23 Feb

Other references · 49

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  4. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
  5. 2025Qwen2.5-VL Technical Reportcited by 5,418
  6. 2024Qwen2.5 Technical Reportcited by 4,713
  7. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  8. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  9. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,769
  10. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,573
  11. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,061
  12. 2023ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generationcited by 1,010
37 more
  1. 2023MVDream: Multi-view Diffusion for 3D Generationcited by 928
  2. 2023Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotationscited by 895
  3. 2023Human Preference Score v2: A Solid Benchmark for Evaluating Human Preferences of Text-to-Image Synthesiscited by 847
  4. 2024Show-o: One Single Transformer to Unify Multimodal Understanding and Generationcited by 734
  5. 2025Qwen2.5-Omni Technical Reportcited by 704
  6. 2025Vision-R1: Incentivizing Reasoning Capability in Multimodal Large Language Modelscited by 653
  7. 2023NExT-GPT: Any-to-Any Multimodal LLMcited by 610
  8. 2024RewardBench: Evaluating Reward Models for Language Modelingcited by 480
  9. 2024MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmarkcited by 362
  10. 2024Skywork-Reward: Bag of Tricks for Reward Modeling in LLMscited by 322
  11. 2024ProcessBench: Identifying Process Errors in Mathematical Reasoningcited by 277
  12. 2024RLHF Workflow: From Reward Modeling to Online RLHFcited by 265
  13. 2025Inference-Time Scaling for Generalist Reward Modelingcited by 233
  14. 2024HelpSteer2: Open-source dataset for training top-performing reward modelscited by 232
  15. 2023Listen, Think, and Understandcited by 206
  16. 2024VideoScore: Building Automatic Metrics to Simulate Fine-grained Human Feedback for Video Generationcited by 190
  17. 2024LLaMA-Omni: Seamless Speech Interaction with Large Language Modelscited by 169
  18. 2024RM-Bench: Benchmarking Reward Models of Language Models with Subtlety and Stylecited by 168
  19. 2025Step-Video-T2V Technical Report: The Practice, Challenges, and Future of Video Foundation Modelcited by 163
  20. 2023Text-to-Audio Generation using Instruction-Tuned LLM and Latent Diffusion Modelcited by 145
  21. 2024LLaVA-Critic: Learning to Evaluate Multimodal Modelscited by 132
  22. 2024Tango 2: Aligning Diffusion-based Text-to-Audio Generations through Direct Preference Optimizationcited by 97
  23. 2024OffsetBias: Leveraging Debiased Data for Tuning Evaluatorscited by 94
  24. 2024Cutting Off the Head Ends the Conflict: A Mechanism for Interpreting and Mitigating Knowledge Conflicts in Language Modelscited by 87
  25. 2024Aligning to Thousands of Preferences via System Message Generalizationcited by 84
  26. 2024GenAI Arena: An Open Evaluation Platform for Generative Modelscited by 81
  27. 2024RWKU: Benchmarking Real-World Knowledge Unlearning for Large Language Modelscited by 77
  28. 2023Ties Matter: Meta-Evaluating Modern Metrics with Pairwise Accuracy and Tie Calibrationcited by 71
  29. 2024MIA-Bench: Towards Better Instruction Following Evaluation of Multimodal LLMscited by 62
  30. 2024DreamReward: Text-to-3D Generation with Human Preferencecited by 56
  31. 2024Align Anything: Training All-Modality Models to Follow Instructions with Language Feedbackcited by 38
  32. 2024EvalMuse-40K: A Reliable and Fine-Grained Benchmark with Comprehensive Human Annotations for Text-to-Image Generation Model Evaluationcited by 37
  33. 2024Towards Robust Knowledge Unlearning: An Adversarial Framework for Assessing and Improving Unlearning Robustness in Large Language Modelscited by 24
  34. 2024Improving Context-Aware Preference Modeling for Language Modelscited by 22
  35. 2024RAG-RewardBench: Benchmarking Reward Models in Retrieval Augmented Generation for Preference Alignmentcited by 22
  36. 2025MMR-V: What's Left Unsaid? A Benchmark for Multimodal Deep Reasoning in Videoscited by 22
  37. 2025Agent-RewardBench: Towards a Unified Benchmark for Reward Modeling across Perception, Planning, and Safety in Real-World Multimodal Agentscited by 19

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 10.

11 papers.

  1. 2026UniCorn: Towards Self-Improving Unified Multimodal Models through Self-Generated Supervisioncited by 23
  2. 2026OmniGAIA: Towards Native Omni-Modal AI Agentscited by 21
  3. 2025UI2Code^N: UI-to-Code Generation as Interactive Visual Optimizationcited by 12
  4. 2026MMR-Life: Piecing Together Real-life Scenes for Multimodal Multi-image Reasoningcited by 8
  5. 2026Agentic Environment Engineering for Large Language Models: A Survey of Environment Modeling, Synthesis, Evaluation, and Applicationcited by 8
  6. 2026Omni-RRM: Advancing Omni Reward Modeling via Automatic Rubric-Grounded Preference Synthesiscited by 7
  7. 2026Beyond Simply Environment Scaling: Designing Effective Environment Distributions for Multimodal Agent Learningcited by 2
  8. 2026SFT Conflicts, RL Coexists: A Theoretical and Empirical Analysis of Multi-Task Learning for LLMscited by 1
  9. 2026DT2IT-MRM: Debiased Preference Construction and Iterative Training for Multimodal Reward Modelingcited by 0
  10. 2026Video Understanding Reward Modeling: A Robust Benchmark and Performant Reward Modelscited by 0
  11. 2026OmniJudge or OmniBias? Diagnosing Multimodal Judges through Balanced, Decoupled Lensescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Multimodal RewardBench 2: Evaluating Omni Reward Models for Interleaved Text and Image
  2. 2025Activation Reward Models for Few-Shot Model Alignment
  3. 2026Omni-Perception Policy Optimization for Multimodal Emotion Reasoning
  4. 2025BaseReward: A Strong Baseline for Multimodal Reward Model
  5. 2025R1-Omni: Explainable Omni-Multimodal Emotion Recognition with Reinforcement Learning
  6. 2025Skywork-Reward-V2: Scaling Preference Data Curation via Human-AI Synergy
  7. 2026Personalized RewardBench: Evaluating Reward Models with Human Aligned Personalization
  8. 2025The Devil Is in the Details: Tackling Unimodal Spurious Correlations for Generalizable Multimodal Reward Models