mlsanityData to 7 Oct 2026

Boost Your Human Image Generation Model via Direct Preference Optimization

arXiv 2405.20216 ↗First posted 30 May 2024cs.CV cs.AI cs.LGcited by 16

On the map: LLM alignment / RLHF / preference optimisation → Diffusion and multimodal model alignment

Abstract (arXiv)

Human image generation is a key focus in image synthesis due to its broad applications, but even slight inaccuracies in anatomy, pose, or details can compromise realism. To address these challenges, we explore Direct Preference Optimization (DPO), which trains models to generate preferred (winning) images while diverging from non-preferred (losing) ones. However, conventional DPO methods use generated images as winning images, limiting realism. To overcome this limitation, we propose an enhanced DPO approach that incorporates high-quality real images as winning images, encouraging outputs to resemble real images rather than generated ones. However, implementing this concept is not a trivial task. Therefore, our approach, HG-DPO (Human image Generation through DPO), employs a novel curriculum learning framework that gradually improves the output of the model toward greater realism, making training more feasible. Furthermore, HG-DPO effectively adapts to personalized text-to-image tasks, generating high-quality and identity-specific images, which highlights the practical value of our approach.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Placed in Diffusion and multimodal model alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 66 of its 91 references that are in the map. First posted on arXiv 30 May 2024.

Builds on

66 of 91 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024Curriculum Direct Preference Optimization for Diffusion and Consistency Models22 May
  2. 2024Self-Play Fine-Tuning of Diffusion Models for Text-to-Image Generation15 Feb
  3. 2023Using Human Feedback to Fine-tune Diffusion Models without Any Reward Model22 Nov
  4. 2023Diffusion Model Alignment Using Direct Preference Optimization21 Nov
  5. 2023Directly Fine-Tuning Diffusion Models on Differentiable Rewards29 Sep

Other references · 61

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2022Training language models to follow instructions with human feedbackcited by 15,712
  5. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  6. 2020Denoising Diffusion Implicit Modelscited by 9,655
  7. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  8. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  9. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  10. 2023Visual Instruction Tuningcited by 7,792
  11. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  12. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
38 more
  1. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 5,986
  2. 2022Classifier-Free Diffusion Guidancecited by 5,717
  3. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
  4. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  5. 2019Analyzing and Improving the Image Quality of StyleGANcited by 3,580
  6. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,286
  7. 2022DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Stepscited by 1,961
  8. 2021SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equationscited by 1,903
  9. 2021Alias-Free Generative Adversarial Networkscited by 1,181
  10. 2022Exploring CLIP for Assessing the Look and Feel of Imagescited by 974
  11. 2023Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generationcited by 950
  12. 2017VGGFace2: A dataset for recognising faces across pose and agecited by 911
  13. 2023Human Preference Score v2: A Solid Benchmark for Evaluating Human Preferences of Text-to-Image Synthesiscited by 847
  14. 2021Vector Quantized Diffusion Model for Text-to-Image Synthesiscited by 773
  15. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 630
  16. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 552
  17. 2021FlexMatch: Boosting Semi-Supervised Learning with Curriculum Pseudo Labelingcited by 537
  18. 2023DPOK: Reinforcement Learning for Fine-tuning Text-to-Image Diffusion Modelscited by 431
  19. 2023AnyDoor: Zero-shot Object-level Image Customizationcited by 411
  20. 2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Trainingcited by 399
  21. 20218-bit Optimizers via Block-wise Quantizationcited by 395
  22. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedbackcited by 395
  23. 2023InstantBooth: Personalized Text-to-Image Generation without Test-Time Finetuningcited by 343
  24. 2023Statistical Rejection Sampling Improves Preference Optimizationcited by 321
  25. 2023FastComposer: Tuning-Free Multi-Subject Image Generation with Localized Attentioncited by 313
  26. 2023Aligning Text-to-Image Diffusion Models with Reward Backpropagationcited by 284
  27. 2023Preference Ranking Optimization for Human Alignmentcited by 283
  28. 2023Pretraining Language Models with Human Preferencescited by 245
  29. 2023Subject-Diffusion:Open Domain Personalized Text-to-Image Generation without Test-time Fine-tuningcited by 191
  30. 2017Reverse Curriculum Generation for Reinforcement Learningcited by 185
  31. 2024LiPO: Listwise Preference Optimization through Learning-to-Rankcited by 94
  32. 2020PLATO-2: Towards Building an Open-Domain Chatbot via Curriculum Learningcited by 79
  33. 2023DisenBooth: Identity-Preserving Disentangled Tuning for Subject-Driven Text-to-Image Generationcited by 75
  34. 2023Aligning Large Language Models with Human Preferences through Representation Engineeringcited by 71
  35. 2017Curriculum Learning and Minibatch Bucketing in Neural Machine Translationcited by 63
  36. 2020Norm-Based Curriculum Learning for Neural Machine Translationcited by 60
  37. 2019Curriculum Self-Paced Learning for Cross-Domain Object Detectioncited by 39
  38. 2023Enhancing Diffusion Models with Text-Encoder Reinforcement Learningcited by 30

50 most cited of 61 All 61 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 5; 2026: 11.

16 papers.

Sort:

By area:

  1. 2025PhyGDPO: Physics-Aware Groupwise Direct Preference Optimization for Physically Consistent Text-to-Video Generationcited by 22
  2. 2025Integrating Reinforcement Learning with Visual Generative Models: Foundations and Advancescited by 11
  3. 2025MatAnyone 2: Scaling Video Matting via a Learned Quality Evaluatorcited by 7
  4. 2026TP-Blend: Textual-Prompt Attention Pairing for Precise Object-Style Blending in Diffusion Modelscited by 4
  5. 2025Ranking-based Preference Optimization for Diffusion Models from Implicit User Feedbackcited by 3
  6. 2025Reg-DPO: SFT-Regularized Direct Preference Optimization with GT-Pair for Improving Video Generationcited by 2
  7. 2026Human detectors are surprisingly powerful reward modelscited by 1
  8. 2026SkeleGuide: Explicit Skeleton Reasoning for Context-Aware Human-in-Place Image Synthesiscited by 1
  9. 2026POCA: Pareto-Optimal Curriculum Alignment for Visual Text Generationcited by 1
  10. 2026Offline Preference Optimization for Rectified Flow with Noise-Tracked Pairscited by 1
  11. 2026Towards Anatomically Plausible Human Image Generation via Synthetic Localized Preferencescited by 1
  12. 2026Curriculum-DPO++: Direct Preference Optimization via Data and Model Curricula for Text-to-Image Generationcited by 0
4 more
  1. 2026Hierarchical Image Tokenization for Multi-Scale Image Super Resolutioncited by 0
  2. 2026SeamCam: Quantifying Seamless Camouflage via Multi-Cue Visual Detectabilitycited by 0
  3. 2026DivRL: Disentangled Self-Similarity Rewards for Diverse Subject-Driven Generationcited by 0
  4. 2026CMA-OT: Hierarchical Expert Supervision for Dance-to-Music Generationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024SEE-DPO: Self Entropy Enhanced Direct Preference Optimization
  2. 2025HuViDPO:Enhancing Video Generation through Direct Preference Optimization for Human-Centric Alignment
  3. 2024Scalable Ranked Preference Optimization for Text-to-Image Generation
  4. 2025When Preferences Diverge: Aligning Diffusion Models with Minority-Aware Adaptive DPO
  5. 2025DSPO: Direct Semantic Preference Optimization for Real-World Image Super-Resolution
  6. 2024Aesthetic Post-Training Diffusion Models from Generic Preferences with Step-by-step Preference Optimization
  7. 2025Pref-GRPO: Pairwise Preference Reward-based GRPO for Stable Text-to-Image Reinforcement Learning
  8. 2025AR-GRPO: Training Autoregressive Image Generation Models via Reinforcement Learning