Refining Multidimensional Video Reward Models via Disentangled Influence Functions
On the map: LLM alignment / RLHF / preference optimisation → Diffusion and multimodal model alignment
Abstract (arXiv)
As Text-to-Video (T2V) generation models continue to evolve, the complexity of video evaluation necessitates a fine-grained assessment across various axes. To address this, recent works have focused on developing Multidimensional Video Reward Models (MVRMs), which decompose the evaluation process to better align with the multifaceted nature of human visual perception. However, training effective MVRMs is fundamentally challenged by the complex nature of video data. In this work, we identify a critical phenomenon termed Dimensional Heterogeneity: the reliability of a training sample can vary substantially across evaluation dimensions, meaning that a sample may provide reliable supervision for one objective while inducing high supervision risk for another. Consequently, prevailing data-centric methods that filter based on global scalar metrics are ill-posed for T2V tasks. To address this, we propose a disentangled influence framework that that efficiently estimates dimension-specific supervision risk. Leveraging this framework, we introduce two dimension-disentangled refinement strategies: Dimension-Disentangled Pruning, which removes extreme high-risk samples, and Dimension-Disentangled Reweighting, which softly down-weights high-risk supervision. Extensive experiments demonstrate that our disentangled strategies significantly outperform global filtering baselines, yielding reward models with superior alignment to ground truth.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Diffusion and multimodal model alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 30 of its 33 references that are in the map. First posted on arXiv 27 May 2026.
Builds on
30 of 33 references.
Closest ancestors · 2
Same direction, newest first
- 2025Improving Video Generation with Human Feedback23 Jan
- 2024VisionReward: Fine-Grained Multi-Dimensional Human Preference Learning for Image and Video Generation30 Dec
Other references · 28
- 2017Deep reinforcement learning from human preferencescited by 3,846
- 2020Learning to summarize from human feedbackcited by 2,562
- 2025Wan: Open and Advanced Large-Scale Video Generative Modelscited by 2,536
- 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,526
- 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,388
- 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
- 2023VBench: Comprehensive Benchmark Suite for Video Generative Modelscited by 1,755
- 2017Understanding Black-box Predictions via Influence Functionscited by 1,714
- 2024HunyuanVideo: A Systematic Framework For Large Video Generative Modelscited by 1,597
- 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,385
- 2017Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semanticscited by 1,345
- 2020Gradient Surgery for Multi-Task Learningcited by 1,210
16 more
- 2018Towards Accurate Generative Models of Video: A New Metric & Challengescited by 1,204
- 2022CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformerscited by 973
- 2023ModelScope Text-to-Video Technical Reportcited by 651
- 2020Estimating Training Data Influence by Tracing Gradient Descentcited by 565
- 2023EvalCrafter: Benchmarking and Evaluating Large Video Generation Modelscited by 332
- 2023TRAK: Attributing Model Behavior at Scalecited by 273
- 2024MANTIS: Interleaved Multi-Image Instruction Tuningcited by 212
- 2024VideoScore: Building Automatic Metrics to Simulate Fine-grained Human Feedback for Video Generationcited by 190
- 2018Fast Approximate Natural Gradient Descent in a Kronecker-factored Eigenbasiscited by 143
- 2023FETV: A Benchmark for Fine-Grained Evaluation of Open-Domain Text-to-Video Generationcited by 119
- 2021Reasonable Effectiveness of Random Weighting: A Litmus Test for Multi-Task Learningcited by 100
- 2024GenAI Arena: An Open Evaluation Platform for Generative Modelscited by 81
- 2023Data Pruning via Moving-one-Sample-outcited by 67
- 2024Variational Bayesian Last Layerscited by 67
- 2025Kling-Omni Technical Reportcited by 60
- 2025Understanding Impact of Human Feedback via Influence Functionscited by 11
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025SoliReward: Mitigating Susceptibility to Reward Hacking and Annotation Noise in Video Generation Reward Models
- 2026Reward Lightning: Fast Video Generation via Homologous Preference Distillation
- 2026Aligning Human Sense: Calibrated Distributional Reward Learning for Video Generation
- 2025VideoScore2: Think before You Score in Generative Video Evaluation
- 2026CoRe: Co-Evolving Reward Models for Mitigating Latent Reward Hacking in Video Diffusion Models
- 2026Thinking with Frames: Generative Video Distortion Evaluation via Frame Reward Model
- 2026Video Understanding Reward Modeling: A Robust Benchmark and Performant Reward Models
- 2024Video Diffusion Alignment via Reward Gradients