mlsanityData to 14 Oct 2026

MJ-VIDEO: Fine-Grained Benchmarking and Rewarding Video Preferences in Video Generation

arXiv 2502.01719 ↗First posted 3 Feb 2025cs.CVcited by 16

On the map: Mixture-of-experts → Vision mixture-of-experts

Abstract (arXiv)

Recent advancements in video generation have significantly improved the ability to synthesize videos from text instructions. However, existing models still struggle with key challenges such as instruction misalignment, content hallucination, safety concerns, and bias. Addressing these limitations, we introduce MJ-BENCH-VIDEO, a large-scale video preference benchmark designed to evaluate video generation across five critical aspects: Alignment, Safety, Fineness, Coherence & Consistency, and Bias & Fairness. This benchmark incorporates 28 fine-grained criteria to provide a comprehensive evaluation of video preference. Building upon this dataset, we propose MJ-VIDEO, a Mixture-of-Experts (MoE)-based video reward model designed to deliver fine-grained reward. MJ-VIDEO can dynamically select relevant experts to accurately judge the preference based on the input text-video pair. This architecture enables more precise and adaptable preference judgments. Through extensive benchmarking on MJ-BENCH-VIDEO, we analyze the limitations of existing video reward models and demonstrate the superior performance of MJ-VIDEO in video preference assessment, achieving 17.58% and 15.87% improvements in overall and fine-grained preference judgments, respectively. Additionally, introducing MJ-VIDEO for preference tuning in video generation enhances the alignment performance. All our code, data, and models are available at https://aiming-lab.github.io/MJ-VIDEO.github.io/.

Key facts. Cited by 16 papers in the map (data through 14 Oct 2026). Placed in Vision mixture-of-experts, one of 12 directions in Mixture-of-experts. Builds on 47 of its 50 references that are in the map. First posted on arXiv 3 Feb 2025.

Builds on

47 of 50 references.

References · 47

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  3. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  4. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,581
  5. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
  6. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  7. 2023VBench: Comprehensive Benchmark Suite for Video Generative Modelscited by 1,822
  8. 2018Towards Accurate Generative Models of Video: A New Metric & Challengescited by 1,234
  9. 2022CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformerscited by 985
  10. 2023Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generationcited by 977
  11. 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 927
  12. 2023Human Preference Score v2: A Solid Benchmark for Evaluating Human Preferences of Text-to-Image Synthesiscited by 890
35 more
  1. 2023Diffusion Model Alignment Using Direct Preference Optimizationcited by 790
  2. 2024Open-Sora: Democratizing Efficient Video Production for Allcited by 652
  3. 2022Mixture-of-Experts with Expert Choice Routingcited by 623
  4. 2021VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understandingcited by 602
  5. 2024VideoCrafter2: Overcoming Data Limitations for High-Quality Video Diffusion Modelscited by 579
  6. 2023DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Modelscited by 518
  7. 2023VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Taskscited by 501
  8. 2024Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Expertscited by 376
  9. 2024MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmarkcited by 372
  10. 2024OpenVid-1M: A Large-Scale High-Quality Dataset for Text-to-video Generationcited by 341
  11. 2020FineGym: A Hierarchical Video Dataset for Fine-grained Action Understandingcited by 234
  12. 2022DALL-Eval: Probing the Reasoning Skills and Social Biases of Text-to-Image Generation Modelscited by 230
  13. 2021Associating Objects with Transformers for Video Object Segmentationcited by 226
  14. 2024CogVLM2: Visual Language Models for Image and Video Understandingcited by 217
  15. 2024VideoScore: Building Automatic Metrics to Simulate Fine-grained Human Feedback for Video Generationcited by 197
  16. 2024HALC: Object Hallucination Reduction via Adaptive Focal-Contrast Decodingcited by 174
  17. 2024VisionReward: Fine-Grained Multi-Dimensional Human Preference Learning for Image and Video Generationcited by 156
  18. 2024VidProM: A Million-scale Real Prompt-Gallery Dataset for Text-to-Video Diffusion Modelscited by 131
  19. 2024GRAPE: Generalizing Robot Policy via Preference Alignmentcited by 109
  20. 2024Video Diffusion Alignment via Reward Gradientscited by 107
  21. 2024Mementos: A Comprehensive Benchmark for Multimodal Large Language Model Reasoning over Image Sequencescited by 94
  22. 2023InstructVideo: Instructing Video Diffusion Models with Human Feedbackcited by 86
  23. 2024GenAI Arena: An Open Evaluation Platform for Generative Modelscited by 82
  24. 2023Video Prediction Models as Rewards for Reinforcement Learningcited by 78
  25. 2024Calibrated Self-Rewarding Vision Language Modelscited by 78
  26. 2024MJ-Bench: Is Your Multimodal Reward Model Really a Good Judge for Text-to-Image Generation?cited by 66
  27. 2024VidGen-1M: A Large-Scale Dataset for Text-to-video Generationcited by 62
  28. 2023Diffusion Reward: Learning Rewards via Conditional Video Diffusioncited by 57
  29. 2021One TTS Alignment To Rule Them Allcited by 47
  30. 2024SafeWatch: An Efficient Safety-Policy Following Video Guardrail Model with Transparent Explanationscited by 38
  31. 2024SafeSora: Towards Safety Alignment of Text2Video Generation via a Human Preference Datasetcited by 35
  32. 2024AutoPRM: Automating Procedural Supervision for Multi-Step Reasoning via Controllable Question Decompositioncited by 24
  33. 2024Fine-Grained Verifiers: Preference Modeling as Next-token Prediction in Vision-Language Alignmentcited by 19
  34. 2024Sora Detector: A Unified Hallucination Detection for Large Text-to-Video Modelscited by 17
  35. 2024Preference Optimization with Multi-Sample Comparisonscited by 16

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 7; 2026: 9.

16 papers.

Sort:

By area:

  1. 2025Thinking with Images for Multimodal Reasoning: Foundations, Methods, and Future Frontierscited by 199
  2. 2025MDocAgent: A Multi-Modal Multi-Agent Framework for Document Understandingcited by 67
  3. 2025VideoScore2: Think before You Score in Generative Video Evaluationcited by 40
  4. 2025T2AV-Compass: Towards Unified Evaluation for Text-to-Audio-Video Generationcited by 24
  5. 2026Unified Personalized Reward Model for Vision Generationcited by 13
  6. 2025VR-Thinker: Boosting Video Reward Models through Thinking-with-Image Reasoningcited by 12
  7. 2026PhyGround: Benchmarking Physical Reasoning in Generative World Modelscited by 12
  8. 2025Anyprefer: An Agentic Framework for Preference Data Synthesiscited by 10
  9. 2025McSc: Motion-Corrective Preference Alignment for Video Generation with Self-Critic Hierarchical Reasoningcited by 2
  10. 2026FIRM-Video: Check Before You Score for Reliable Text-to-Video Reward Modelingcited by 2
  11. 2026VGIF-Score: Interpretable and Diagnostic Evaluation of Spatio-Temporal Instruction Following in Video Generationcited by 1
  12. 2026RewardVerse: Rubric-Guided Policy Optimization for Video Reward Modelingcited by 1
4 more
  1. 2026Analytic Score Optimization for Multi Dimension Video Quality Assessmentcited by 0
  2. 2026Plan-and-Verify Video Reward Reasoning with Spatio-Temporal Scene Graph Groundingcited by 0
  3. 2026GraftOOD: Grafting Out-of-Distribution Exemplars into Advantage-Weighted Rectification for Video Reward Optimizationcited by 0
  4. 2026IsoVRBench: Video Reward Modeling Without Human Feedbackcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Improving Video Generation with Human Feedback
  2. 2025Dual-IPO: Dual-Iterative Preference Optimization for Text-to-Video Generation
  3. 2026Video Understanding Reward Modeling: A Robust Benchmark and Performant Reward Models
  4. 2025SoliReward: Mitigating Susceptibility to Reward Hacking and Annotation Noise in Video Generation Reward Models
  5. 2025Video-Bench: Human-Aligned Video Generation Benchmark
  6. 2026Aligning Human Sense: Calibrated Distributional Reward Learning for Video Generation
  7. 2026KeyFrame-Compass: Towards Comprehensive Evaluation of Keyframe-Conditioned Video Generation
  8. 2026VideoArgus: Agentic Rubric-Grounded Unified Evaluation for Video Generation and Editing