mlsanityData to 7 Oct 2026

LiFT: Leveraging Human Feedback for Text-to-Video Model Alignment

arXiv 2412.04814 ↗First posted 6 Dec 2024cs.CVcited by 51

On the map: LLM alignment / RLHF / preference optimisation → Diffusion and multimodal model alignment

Abstract (arXiv)

Recent advances in text-to-video (T2V) generative models have shown impressive capabilities. However, these models are still inadequate in aligning synthesized videos with human preferences (e.g., accurately reflecting text descriptions), which is particularly difficult to address, as human preferences are subjective and challenging to formalize as objective functions. Existing studies train video quality assessment models that rely on human-annotated ratings for video evaluation but overlook the reasoning behind evaluations, limiting their ability to capture nuanced human criteria. Moreover, aligning T2V model using video-based human feedback remains unexplored. Therefore, this paper proposes LiFT, the first method designed to leverage human feedback for T2V model alignment. Specifically, we first construct a Human Rating Annotation dataset, LiFT-HRA, consisting of approximately 10k human annotations, each including a score and its corresponding rationale. Based on this, we train a reward model LiFT-Critic to learn reward function effectively, which serves as a proxy for human judgment, measuring the alignment between given videos and human expectations. Lastly, we leverage the learned reward function to align the T2V model by maximizing the reward-weighted likelihood. As a case study, we apply our pipeline to CogVideoX-2B, showing that the fine-tuned model outperforms the CogVideoX-5B across all 16 metrics, highlighting the potential of human feedback in improving the alignment and quality of synthesized videos.

Key facts. Cited by 51 papers in the map (data through 7 Oct 2026). Placed in Diffusion and multimodal model alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 37 of its 39 references that are in the map. First posted on arXiv 6 Dec 2024.

Builds on

37 of 39 references.

Closest ancestors · 8

Same direction, newest first

  1. 2024T2V-Turbo-v2: Enhancing Video Generation Model Post-Training through Data, Reward, and Conditional Guidance Design8 Oct
  2. 2024Video Diffusion Alignment via Reward Gradients11 Jul
  3. 2024T2V-Turbo: Breaking the Quality Bottleneck of Video Consistency Model with Mixed Reward Feedback29 May
  4. 2024Reward Guided Latent Consistency Distillation16 Mar
  5. 2023InstructVideo: Instructing Video Diffusion Models with Human Feedback19 Dec
  6. 2023Rich Human Feedback for Text-to-Image Generation15 Dec
  7. 2023Directly Fine-Tuning Diffusion Models on Differentiable Rewards29 Sep
  8. 2023Aligning Text-to-Image Models using Human Feedback23 Feb

Other references · 29

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  5. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  6. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  7. 2024CogVideoX: Text-to-Video Diffusion Models with An Expert Transformercited by 2,198
  8. 2024Qwen2 Technical Reportcited by 2,076
  9. 2023VBench: Comprehensive Benchmark Suite for Video Generative Modelscited by 1,755
  10. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Modelscited by 1,465
  11. 2023AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuningcited by 1,418
  12. 2023Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generationcited by 950
17 more
  1. 2023Training Diffusion Models with Reinforcement Learningcited by 939
  2. 2023Human Preference Score v2: A Solid Benchmark for Evaluating Human Preferences of Text-to-Image Synthesiscited by 847
  3. 2023VILA: On Pre-training for Visual Language Modelscited by 686
  4. 2023ModelScope Text-to-Video Technical Reportcited by 651
  5. 2023VideoCrafter1: Open Diffusion Models for High-Quality Video Generationcited by 530
  6. 2023VideoComposer: Compositional Video Synthesis with Motion Controllabilitycited by 461
  7. 2024OpenVid-1M: A Large-Scale High-Quality Dataset for Text-to-video Generationcited by 332
  8. 2023Photorealistic Video Generation with Diffusion Modelscited by 304
  9. 2024VideoScore: Building Automatic Metrics to Simulate Fine-grained Human Feedback for Video Generationcited by 190
  10. 2023AGIQA-3K: An Open Database for AI-Generated Image Quality Assessmentcited by 142
  11. 2024Learning Multi-dimensional Human Preference for Text-to-Image Generationcited by 102
  12. 2024Large-scale Reinforcement Learning for Diffusion Modelscited by 78
  13. 2024Allegro: Open the Black Box of Commercial-Level Video Generation Modelcited by 68
  14. 2024VEnhancer: Generative Space-Time Enhancement for Video Generationcited by 64
  15. 2024VidGen-1M: A Large-Scale Dataset for Text-to-video Generationcited by 60
  16. 2024Subjective-Aligned Dataset and Metric for Text-to-Video Quality Assessmentcited by 39
  17. 2023Enhancing Diffusion Models with Text-Encoder Reinforcement Learningcited by 30

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 30; 2026: 21.

51 papers.

Sort:

By area:

  1. 2025Improving Video Generation with Human Feedbackcited by 267
  2. 2025Unified Reward Model for Multimodal Understanding and Generationcited by 194
  3. 2025Unified Multimodal Chain-of-Thought Reward Model through Reinforcement Fine-Tuningcited by 86
  4. 2025VideoScore2: Think before You Score in Generative Video Evaluationcited by 37
  5. 2025DenseDPO: Fine-Grained Temporal Preference Optimization for Video Diffusion Modelscited by 32
  6. 2025VQ-Insight: Teaching VLMs for AI-Generated Video Quality Understanding via Progressive Visual Reinforcement Learningcited by 26
  7. 2025UniGenBench++: A Unified Semantic Evaluation Benchmark for Text-to-Image Generationcited by 23
  8. 2025UniREditBench: A Unified Reasoning-based Image Editing Benchmarkcited by 23
  9. 2025FingER: Content Aware Fine-grained Evaluation with Reasoning for AI-Generated Videoscited by 22
  10. 2025HumanAesExpert: Advancing a Multi-Modality Foundation Model for Human Image Aesthetic Assessmentcited by 17
  11. 2025Sailing by the Stars: A Survey on Reward Models and Learning Strategies for Learning from Rewardscited by 14
  12. 2025IndustryEQA: Pushing the Frontiers of Embodied Question Answering in Industrial Scenarioscited by 12
38 more
  1. 2025Integrating Reinforcement Learning with Visual Generative Models: Foundations and Advancescited by 11
  2. 2025VR-Thinker: Boosting Video Reward Models through Thinking-with-Image Reasoningcited by 11
  3. 2025Omni-Reward: Towards Generalist Omni-Modal Reward Modeling with Free-Form Preferencescited by 11
  4. 2026Unified Personalized Reward Model for Vision Generationcited by 11
  5. 2025Identity-GRPO: Optimizing Multi-Human Identity-preserving Video Generation via Reinforcement Learningcited by 9
  6. 2025RAPO++: Cross-Stage Prompt Optimization for Text-to-Video Generation via Data Alignment and Test-Time Scalingcited by 9
  7. 2025Aligning Anime Video Generation with Human Feedbackcited by 8
  8. 2025VCapsBench: A Large-scale Fine-grained Benchmark for Video Caption Quality Evaluationcited by 8
  9. 2026Diffusion-DRF: Free, Rich, and Differentiable Reward for Video Diffusion Fine-Tuningcited by 8
  10. 2025RealDPO: Real or Not Real, that is the Preferencecited by 6
  11. 2026Reward Modeling for Reinforcement Learning-Based LLM Reasoning: Design, Challenges, and Evaluationcited by 5
  12. 2026SPIRAL: Self-Evolving Action-Conditioned Video Generation via Reflective Planning Agentscited by 5
  13. 2026MSRL: Scaling Generative Multimodal Reward Modeling via Multi-Stage Reinforcement Learningcited by 5
  14. 2025PhyDetEx: Detecting and Explaining the Physical Plausibility of T2V Modelscited by 4
  15. 2025FantasyTalking2: Timestep-Layer Adaptive Preference Optimization for Audio-Driven Portrait Animationcited by 3
  16. 2025Q-Save: Towards Scoring and Attribution for Generated Video Evaluationcited by 3
  17. 2025SoliReward: Mitigating Susceptibility to Reward Hacking and Annotation Noise in Video Generation Reward Modelscited by 3
  18. 2026Mind the Generative Details: Direct Localized Detail Preference Optimization for Video Diffusion Modelscited by 3
  19. 2026CaC: Advancing Video Reward Models via Hierarchical Spatiotemporal Concentratingcited by 3
  20. 2026VA-Judger: Reward Modeling from Human Preference Feedback for Joint Video-Audio Generationcited by 3
  21. 2025Photography Perspective Composition: Towards Aesthetic Perspective Recommendationcited by 2
  22. 2025Reg-DPO: SFT-Regularized Direct Preference Optimization with GT-Pair for Improving Video Generationcited by 2
  23. 2026Thinking with Frames: Generative Video Distortion Evaluation via Frame Reward Modelcited by 2
  24. 2026Think, then Score: Decoupled Reasoning and Scoring for Video Reward Modelingcited by 2
  25. 2025McSc: Motion-Corrective Preference Alignment for Video Generation with Self-Critic Hierarchical Reasoningcited by 1
  26. 2025Rethinking Prompt Design for Inference-time Scaling in Text-to-Visual Generationcited by 1
  27. 2025EtCon: Edit-then-Consolidate for Reliable Knowledge Editingcited by 1
  28. 2026GT-SVJ: Generative-Transformer-Based Self-Supervised Video Judge For Efficient Video Reward Modelingcited by 1
  29. 2026DRM: Diffusion-based Reward Model With Step-wise Guidancecited by 1
  30. 2025GigaVideo-1: Advancing Video Generation via Automatic Feedback with 4 GPU-Hours Fine-Tuningcited by 0
  31. 2026Sound Sparks Motion: Audio and Text Tuning for Video Editingcited by 0
  32. 2026Plan-and-Verify Video Reward Reasoning with Spatio-Temporal Scene Graph Groundingcited by 0
  33. 2026Through the PRISM: Preference Representation in Intermediate States of Video Diffusion Modelscited by 0
  34. 2026Reward Lightning: Fast Video Generation via Homologous Preference Distillationcited by 0
  35. 2026Temporal Concentration from Rollout Errors: Implicit Preference Optimization for Text-to-Video Diffusioncited by 0
  36. 2026GeoReward: Mitigating Contextual Variable Overestimation in Vision-Language Models for Cross-Market Preference Predictioncited by 0
  37. 2026Manifold Drift in Flow Preference Optimization: A Root Cause of Reward Hackingcited by 0
  38. 2026WorldReward: Reward Modeling for Camera-Conditioned World Modelscited by 0

50 most cited of 51 All 51 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024SafeSora: Towards Safety Alignment of Text2Video Generation via a Human Preference Dataset
  2. 2026Aligning Human Sense: Calibrated Distributional Reward Learning for Video Generation
  3. 2025Video-Bench: Human-Aligned Video Generation Benchmark
  4. 2024Confidence-aware Reward Optimization for Fine-tuning Text-to-Image Models
  5. 2024Improving Dynamic Object Interactions in Text-to-Video Generation with AI Feedback
  6. 2024HumanVBench: Probing Human-Centric Video Understanding in MLLMs with Automatically Synthesized Benchmarks
  7. 2024Towards A Better Metric for Text-to-Video Generation
  8. 2024Free2Guide: Training-Free Text-to-Video Alignment using Image LVLM