mlsanityData to 7 Oct 2026

Omni-RRM: Advancing Omni Reward Modeling via Automatic Rubric-Grounded Preference Synthesis

arXiv 2602.00846 ↗First posted 31 Jan 2026cs.CLcited by 7

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

also in 1 more
Abstract (arXiv)

Multimodal large language models (MLLMs) struggle with alignment due to the limitations of existing reward models (RMs), which are predominantly vision-centric, dependent on costly human labels, and provide opaque scalar scores that fail to capture nuanced reasoning, leading to brittle alignment. We present Omni-RRM, an Omni-modal Rubric-grounded Reward Model that generates multi-dimensional reward signals across text, image, video, and audio. To overcome the high cost and inherent inconsistency of human-centric evaluation in multi-dimensional reasoning, we introduce Omni-Preference, a high-quality dataset constructed via automatic rubric-grounded preference synthesis. In this pipeline, teacher models reconcile raw preferences into explicit justifications, ensuring that the synthesized supervision is both high-fidelity and interpretable. Omni-RRM is trained using a progressive SFT + GRPO regimen, specifically optimized to sharpen reward discrimination on low-margin, hard preference pairs. It achieves state-of-the-art accuracy on video (80.2% on ShareGPT-Video) and audio benchmarks (66.8% on Audio-HH-RLHF and 65.0% on TA2T), yielding a five-benchmark Overall accuracy of 70.4% and a +17.0% relative gain over its backbone. Furthermore, Omni-RRM effectively guides Best-of-N selection and exhibits robust transfer to text-only alignment. All resources, including the dataset, training and inference code, and model checkpoints are available at https://tmfk418.github.io/Omni-RRM.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 47 of its 62 references that are in the map. First posted on arXiv 31 Jan 2026.

Builds on

47 of 62 references.

Closest ancestors · 8

Same direction, newest first

  1. 2025BaseReward: A Strong Baseline for Multimodal Reward Model19 Sep
  2. 2025MM-IFEngine: Towards Multimodal Instruction Following10 Apr
  3. 2025Multimodal RewardBench: Holistic Evaluation of Reward Models for Vision Language Models20 Feb
  4. 2024LLaVA-Critic: Learning to Evaluate Multimodal Models3 Oct
  5. 2024LLaVA-OneVision: Easy Visual Task Transfer6 Aug
  6. 2024Direct Preference Optimization of Video Large Multimodal Models from Language Model Reward1 Apr
  7. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks21 Dec
  8. 2023Aligning Large Multimodal Models with Factually Augmented RLHF25 Sep

Other references · 39

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  5. 2023Visual Instruction Tuningcited by 7,792
  6. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
  7. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  8. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  9. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  10. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,769
  11. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,375
  12. 2023ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generationcited by 1,010
27 more
  1. 2023Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generationcited by 950
  2. 2025Qwen2.5-Omni Technical Reportcited by 704
  3. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 630
  4. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackcited by 615
  5. 2025Video-R1: Reinforcing Video Reasoning in MLLMscited by 464
  6. 2025R1-Onevision: Advancing Generalized Multimodal Reasoning through Cross-Modal Formalizationcited by 364
  7. 2025Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domainscited by 328
  8. 2023A Survey of Reinforcement Learning from Human Feedbackcited by 259
  9. 2024HelpSteer2: Open-source dataset for training top-performing reward modelscited by 232
  10. 2025RM-R1: Reward Modeling as Reasoningcited by 149
  11. 2023Rich Human Feedback for Text-to-Image Generationcited by 139
  12. 2024RLAIF-V: Open-Source AI Feedback Leads to Super GPT-4V Trustworthinesscited by 129
  13. 2025OpenRubrics: Towards Scalable Synthetic Rubric Generation for Reward Modeling and LLM Alignmentcited by 118
  14. 2025Unified Multimodal Chain-of-Thought Reward Model through Reinforcement Fine-Tuningcited by 86
  15. 2024VL-RewardBench: A Challenging Benchmark for Vision-Language Generative Reward Modelscited by 85
  16. 2022Clotho-AQA: A Crowdsourced Dataset for Audio Question Answeringcited by 84
  17. 2025R1-Reward: Training Multimodal Reward Model Through Stable Reinforcement Learningcited by 65
  18. 2025Breaking the Exploration Bottleneck: Rubric-Scaffolded Reinforcement Learning for General LLM Reasoningcited by 56
  19. 2025Chasing the Tail: Effective Rubric-based Reward Modeling for Large Language Model Post-Trainingcited by 46
  20. 2025AdvancedIF: Rubric-Based Benchmarking and Reinforcement Learning for Advancing LLM Instruction Followingcited by 46
  21. 2026RubricHub: A Comprehensive and Highly Discriminative Rubric Dataset via Automated Coarse-to-Fine Generationcited by 42
  22. 2025Auto-Rubric: Learning From Implicit Weights to Explicit Rubrics for Reward Modelingcited by 29
  23. 2025Learning to Generate Structured Output with Schema Reinforcement Learningcited by 27
  24. 2024CARMO: Dynamic Criteria Generation for Context-Aware Reward Modellingcited by 21
  25. 2025Robust Reward Modeling via Causal Rubricscited by 18
  26. 2025VR-Thinker: Boosting Video Reward Models through Thinking-with-Image Reasoningcited by 11
  27. 2025Omni-Reward: Towards Generalist Omni-Modal Reward Modeling with Free-Form Preferencescited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 7.

7 papers.

  1. 2026OmniGAIA: Towards Native Omni-Modal AI Agentscited by 21
  2. 2026Visual Preference Optimization with Rubric Rewardscited by 5
  3. 2026EarlyTom: Early Token Compression Completes Fast Video Understandingcited by 3
  4. 2026Reinforcement Learning with Robust Rubric Rewardscited by 2
  5. 2026WaferSAGE: Large Language Model-Powered Wafer Defect Analysis via Synthetic Data Generation and Rubric-Guided Reinforcement Learningcited by 1
  6. 2026A Primer in Post-Training Reasoning Data: What We Know About How It Workscited by 1
  7. 2026Reinforcement Learning with Evolving Rubrics as Rewards for Audio Reasoningcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2025Multimodal RewardBench 2: Evaluating Omni Reward Models for Interleaved Text and Image
  2. 2025Beyond Monolithic Rewards: A Hybrid and Multi-Aspect Reward Optimization for MLLM Alignment
  3. 2026Auto-Rubric as Reward: From Implicit Preferences to Explicit Multimodal Generative Criteria
  4. 2024RRM: Robust Reward Model Training Mitigates Reward Hacking
  5. 2026Joint Reward Modeling: Internalizing Chain-of-Thought for Efficient Visual Reward Models
  6. 2025InternLM-XComposer2.5-Reward: A Simple Yet Effective Multi-Modal Reward Model
  7. 2026VRM: Teaching Reward Models to Understand Authentic Human Preferences
  8. 2025Generative RLHF-V: Learning Principles from Multi-modal Human Preference