mlsanityData to 7 Oct 2026

VisionReward: Fine-Grained Multi-Dimensional Human Preference Learning for Image and Video Generation

arXiv 2412.21059 ↗First posted 30 Dec 2024cs.CVcited by 148

On the map: LLM alignment / RLHF / preference optimisation → Diffusion and multimodal model alignment

Abstract (arXiv)

Visual generative models have achieved remarkable progress in synthesizing photorealistic images and videos, yet aligning their outputs with human preferences across critical dimensions remains a persistent challenge. Though reinforcement learning from human feedback offers promise for preference alignment, existing reward models for visual generation face limitations, including black-box scoring without interpretability and potentially resultant unexpected biases. We present VisionReward, a general framework for learning human visual preferences in both image and video generation. Specifically, we employ a hierarchical visual assessment framework to capture fine-grained human preferences, and leverages linear weighting to enable interpretable preference learning. Furthermore, we propose a multi-dimensional consistent strategy when using VisionReward as a reward model during preference optimization for visual generation. Experiments show that VisionReward can significantly outperform existing image and video reward models on both machine metrics and human evaluation. Notably, VisionReward surpasses VideoScore by 17.2% in preference prediction accuracy, and text-to-video models with VisionReward achieve a 31.6% higher pairwise win rate compared to the same models using VideoScore. All code and datasets are provided at https://github.com/THUDM/VisionReward.

Key facts. Cited by 148 papers in the map (data through 7 Oct 2026). Placed in Diffusion and multimodal model alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 47 of its 57 references that are in the map. First posted on arXiv 30 Dec 2024.

Builds on

47 of 57 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024MJ-Bench: Is Your Multimodal Reward Model Really a Good Judge for Text-to-Image Generation?5 Jul
  2. 2024Margin-aware Preference Optimization for Aligning Diffusion Models without Reference10 Jun
  3. 2024Deep Reward Supervisions for Tuning Text-to-Image Diffusion Models1 May
  4. 2023Rich Human Feedback for Text-to-Image Generation15 Dec
  5. 2023Diffusion Model Alignment Using Direct Preference Optimization21 Nov
  6. 2023Directly Fine-Tuning Diffusion Models on Differentiable Rewards29 Sep

Other references · 41

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2023GPT-4 Technical Reportcited by 18,132
  5. 2022Training language models to follow instructions with human feedbackcited by 15,712
  6. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  7. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  8. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  9. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
  10. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  11. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  12. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
29 more
  1. 2020Learning to summarize from human feedbackcited by 2,562
  2. 2024CogVideoX: Text-to-Video Diffusion Models with An Expert Transformercited by 2,198
  3. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,765
  4. 2023VBench: Comprehensive Benchmark Suite for Video Generative Modelscited by 1,755
  5. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  6. 2021Variational Diffusion Modelscited by 1,145
  7. 2024ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Toolscited by 1,096
  8. 2023ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generationcited by 1,010
  9. 2022CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformerscited by 973
  10. 2023Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generationcited by 950
  11. 2023Training Diffusion Models with Reinforcement Learningcited by 939
  12. 2023Human Preference Score v2: A Solid Benchmark for Evaluating Human Preferences of Text-to-Image Synthesiscited by 847
  13. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 704
  14. 2021CogView: Mastering Text-to-Image Generation via Transformerscited by 691
  15. 2021Maximum Likelihood Training of Score-Based Diffusion Modelscited by 670
  16. 2024LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Modelscited by 571
  17. 2024VideoCrafter2: Overcoming Data Limitations for High-Quality Video Diffusion Modelscited by 568
  18. 2022Phenaki: Variable Length Video Generation From Open Domain Textual Descriptioncited by 475
  19. 2024Evaluating Text-to-Visual Generation with Image-to-Text Generationcited by 465
  20. 2023DPOK: Reinforcement Learning for Fine-tuning Text-to-Image Diffusion Modelscited by 431
  21. 2024Panda-70M: Captioning 70M Videos with Multiple Cross-Modality Teacherscited by 389
  22. 2024CogVLM2: Visual Language Models for Image and Video Understandingcited by 214
  23. 2024VideoScore: Building Automatic Metrics to Simulate Fine-grained Human Feedback for Video Generationcited by 190
  24. 2023AGIQA-3K: An Open Database for AI-Generated Image Quality Assessmentcited by 142
  25. 2024VidProM: A Million-scale Real Prompt-Gallery Dataset for Text-to-Video Diffusion Modelscited by 124
  26. 2023GPT-4V(ision) as a Generalist Evaluator for Vision-Language Taskscited by 118
  27. 2024Learning Multi-dimensional Human Preference for Text-to-Image Generationcited by 102
  28. 2024GenAI Arena: An Open Evaluation Platform for Generative Modelscited by 81
  29. 2023Ties Matter: Meta-Evaluating Modern Metrics with Pairwise Accuracy and Tie Calibrationcited by 71

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 70; 2026: 78.

148 papers.

Sort:

By area:

  1. 2025DanceGRPO: Unleashing GRPO on Visual Generationcited by 388
  2. 2025Improving Video Generation with Human Feedbackcited by 267
  3. 2025Z-Image: An Efficient Image Generation Foundation Model with Single-Stream Diffusion Transformercited by 247
  4. 2025Unified Reward Model for Multimodal Understanding and Generationcited by 194
  5. 2026Causal Forcing: Autoregressive Diffusion Distillation Done Right for High-Quality Real-Time Interactive Video Generationcited by 159
  6. 2025Unified Multimodal Chain-of-Thought Reward Model through Reinforcement Fine-Tuningcited by 86
  7. 2025Uniworld-V2: Reinforce Image Editing with Diffusion Negative-aware Finetuning and MLLM Implicit Feedbackcited by 78
  8. 2025Reward Forcing: Efficient Streaming Video Generation with Rewarded Distribution Matching Distillationcited by 75
  9. 2025TIIF-Bench: How Does Your T2I Model Follow Your Instructions?cited by 64
  10. 2025Radial Attention: O(nlog n) Sparse Attention with Energy Decay for Long Video Generationcited by 64
  11. 2025RewardDance: Reward Scaling in Visual Generationcited by 63
  12. 2025SLA: Beyond Sparsity in Diffusion Transformers via Fine-Tunable Sparse-Linear Attentioncited by 55
38 more
  1. 2026Causal Forcing++: Scalable Few-Step Autoregressive Diffusion Distillation for Real-Time Interactive Video Generationcited by 44
  2. 2025Distribution Matching Distillation Meets Reinforcement Learningcited by 43
  3. 2025Video-T1: Test-Time Scaling for Video Generationcited by 39
  4. 2025Diffusion Model as a Noise-Aware Latent Reward Model for Step-Level Preference Optimizationcited by 35
  5. 2025DenseDPO: Fine-Grained Temporal Preference Optimization for Video Diffusion Modelscited by 32
  6. 2025Does FLUX Already Know How to Perform Physically Plausible Image Composition?cited by 30
  7. 2025VPO: Aligning Text-to-Video Generation Models with Prompt Optimizationcited by 27
  8. 2025VQ-Insight: Teaching VLMs for AI-Generated Video Quality Understanding via Progressive Visual Reinforcement Learningcited by 26
  9. 2025FingER: Content Aware Fine-grained Evaluation with Reasoning for AI-Generated Videoscited by 22
  10. 2025OneReward: Unified Mask-Guided Image Generation via Multi-Task Human Preference Learningcited by 22
  11. 2026SLA2: Sparse-Linear Attention with Learnable Routing and QATcited by 21
  12. 2026SpargeAttention2: Trainable Sparse Attention via Hybrid Top-k+Top-p Masking and Distillation Fine-Tuningcited by 16
  13. 2025SpeechJudge: Towards Human-Level Judgment for Speech Naturalnesscited by 15
  14. 2025MJ-VIDEO: Fine-Grained Benchmarking and Rewarding Video Preferences in Video Generationcited by 14
  15. 2026Manifold-Aware Exploration for Reinforcement Learning in Video Generationcited by 14
  16. 2025DreamFuse: Adaptive Image Fusion with Diffusion Transformercited by 13
  17. 2025Uni-Layout: Integrating Human Feedback in Unified Layout Generation and Evaluationcited by 13
  18. 2025Unified Multimodal Models as Auto-Encoderscited by 13
  19. 2025Dual-IPO: Dual-Iterative Preference Optimization for Text-to-Video Generationcited by 12
  20. 2025Dual-Process Image Generationcited by 12
  21. 2025Integrating Reinforcement Learning with Visual Generative Models: Foundations and Advancescited by 11
  22. 2025VR-Thinker: Boosting Video Reward Models through Thinking-with-Image Reasoningcited by 11
  23. 2025Omni-Reward: Towards Generalist Omni-Modal Reward Modeling with Free-Form Preferencescited by 11
  24. 2026ThinkRL-Edit: Thinking in Reinforcement Learning for Reasoning-Centric Image Editingcited by 10
  25. 2025Identity-GRPO: Optimizing Multi-Human Identity-preserving Video Generation via Reinforcement Learningcited by 9
  26. 2025ImagerySearch: Adaptive Test-Time Search for Video Generation Beyond Semantic Dependency Constraintscited by 9
  27. 2025Seeing What Matters: Visual Preference Policy Optimization for Visual Generationcited by 9
  28. 2025Aligning Anime Video Generation with Human Feedbackcited by 8
  29. 2025RDPO: Real Data Preference Optimization for Physics Consistency Video Generationcited by 8
  30. 2026Diffusion-DRF: Free, Rich, and Differentiable Reward for Video Diffusion Fine-Tuningcited by 8
  31. 2025DreamCS: Geometry-Aware Text-to-3D Generation with Unpaired 3D Reward Supervisioncited by 7
  32. 2025AlignHuman: Improving Motion and Fidelity via Timestep-Segment Preference Optimization for Audio-Driven Human Animationcited by 7
  33. 2025PSR: Scaling Multi-Subject Personalized Image Generation with Pairwise Subject-Consistency Rewardscited by 7
  34. 2026SALAD: Achieve High-Sparsity Attention via Efficient Linear Attention Tuning for Video Diffusion Transformercited by 7
  35. 2025RealDPO: Real or Not Real, that is the Preferencecited by 6
  36. 2025Half-order Fine-Tuning for Diffusion Model: A Recursive Likelihood Ratio Optimizercited by 5
  37. 2025Stable Cinemetrics : Structured Taxonomy and Evaluation for Professional Video Generationcited by 5
  38. 2025Rethinking Reward Signals in Video GRPO: When Scores Become Targetscited by 5

50 most cited of 148 All 148 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Unified Personalized Reward Model for Vision Generation
  2. 2025Test-Time Reasoning Through Visual Human Preferences with VLMs and Soft Rewards
  3. 2026Personalize at Test Time: Learning User Preferences for Image Generation
  4. 2024RoVRM: A Robust Visual Reward Model Optimized via Auxiliary Textual Preference Data
  5. 2026Threshold-Guided Optimization for Visual Generative Models
  6. 2024MVReward: Better Aligning and Evaluating Multi-View Diffusion Models with Human Preferences
  7. 2024Multimodal Large Language Model is a Human-Aligned Annotator for Text-to-Image Generation
  8. 2025Video Generation Models Are Good Latent Reward Models