mlsanityData to 7 Oct 2026

Aligning Text-to-Image Models using Human Feedback

arXiv 2302.12192 ↗First posted 23 Feb 2023cs.LG cs.AI cs.CVcited by 418

On the map: LLM alignment / RLHF / preference optimisation → Diffusion and multimodal model alignment

Abstract (arXiv)

Deep generative models have shown impressive results in text-to-image synthesis. However, current text-to-image models often generate images that are inadequately aligned with text prompts. We propose a fine-tuning method for aligning such models using human feedback, comprising three stages. First, we collect human feedback assessing model output alignment from a set of diverse text prompts. We then use the human-labeled image-text dataset to train a reward function that predicts human feedback. Lastly, the text-to-image model is fine-tuned by maximizing reward-weighted likelihood to improve image-text alignment. Our method generates objects with specified colors, counts and backgrounds more accurately than the pre-trained model. We also analyze several design choices and find that careful investigations on such design choices are important in balancing the alignment-fidelity tradeoffs. Our results demonstrate the potential for learning from human feedback to significantly improve text-to-image models.

Key facts. Cited by 418 papers in the map (data through 7 Oct 2026). Placed in Diffusion and multimodal model alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 44 of its 50 references that are in the map. First posted on arXiv 23 Feb 2023.

Builds on

44 of 50 references.

References · 44

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  6. 2022Training language models to follow instructions with human feedbackcited by 15,712
  7. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  8. 2017Proximal Policy Optimization Algorithmscited by 14,441
  9. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  10. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  11. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  12. 2022Generative Adversarial Networkscited by 5,603
32 more
  1. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  2. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  3. 2017Deep reinforcement learning from human preferencescited by 3,846
  4. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  5. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,286
  6. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  7. 2020Learning to summarize from human feedbackcited by 2,562
  8. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,388
  9. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,231
  10. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  11. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  12. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  13. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,287
  14. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
  15. 2022Multi-Concept Customization of Text-to-Image Diffusioncited by 1,099
  16. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
  17. 2016Pixel Recurrent Neural Networkscited by 814
  18. 2022Compositional Visual Generation with Composable Diffusion Modelscited by 619
  19. 2021ByT5: Towards a token-free future with pre-trained byte-to-byte modelscited by 504
  20. 2022Training-Free Structured Diffusion Guidance for Compositional Text-to-Image Synthesiscited by 386
  21. 2018Reward learning from human preferences and demonstrations in Ataricited by 314
  22. 2021PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-trainingcited by 298
  23. 2021Recursively Summarizing Books with Human Feedbackcited by 287
  24. 2016An Actor-Critic Algorithm for Sequence Predictioncited by 158
  25. 2017Interactive Learning from Policy-Dependent Human Feedbackcited by 155
  26. 2023Chain of Hindsight Aligns Language Models with Feedbackcited by 147
  27. 2017Deep TAMER: Interactive Agent Shaping in High-Dimensional State Spacescited by 122
  28. 2022Character-Aware Models Improve Visual Text Renderingcited by 90
  29. 2018Can Neural Machine Translation be Improved with User Feedback?cited by 71
  30. 2022Training Language Models with Language Feedbackcited by 48
  31. 2020Learning to Compare for Better Training and Evaluation of Open Domain Natural Language Generation Modelscited by 27
  32. 2019VIFIDEL: Evaluating the Visual Fidelity of Image Descriptionscited by 21

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 72; 2024: 130; 2025: 140; 2026: 76.

418 papers.

Sort:

By area:

  1. 2023ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generationcited by 1,010
  2. 2023Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generationcited by 950
  3. 2023Training Diffusion Models with Reinforcement Learningcited by 939
  4. 2023Human Preference Score v2: A Solid Benchmark for Evaluating Human Preferences of Text-to-Image Synthesiscited by 847
  5. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 738
  6. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 672
  7. 2025Flow-GRPO: Training Flow Matching Models via Online RLcited by 605
  8. 2024RewardBench: Evaluating Reward Models for Language Modelingcited by 480
  9. 2023DPOK: Reinforcement Learning for Fine-tuning Text-to-Image Diffusion Modelscited by 431
  10. 2023Directly Fine-Tuning Diffusion Models on Differentiable Rewardscited by 431
  11. 2023TIFA: Accurate and Interpretable Text-to-Image Faithfulness Evaluation with Question Answeringcited by 416
  12. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraintcited by 346
38 more
  1. 2023LayoutGPT: Compositional Visual Planning and Generation with Large Language Modelscited by 337
  2. 2023Human Preference Score: Better Aligning Text-to-Image Models with Human Preferencecited by 284
  3. 2023Aligning Text-to-Image Diffusion Models with Reward Backpropagationcited by 284
  4. 2025Improving Video Generation with Human Feedbackcited by 267
  5. 2023Mix-of-Show: Decentralized Low-Rank Adaptation for Multi-Concept Customization of Diffusion Modelscited by 261
  6. 2023A Survey of Reinforcement Learning from Human Feedbackcited by 259
  7. 2023T2I-CompBench++: An Enhanced and Comprehensive Benchmark for Compositional Text-to-image Generationcited by 253
  8. 2023Using Human Feedback to Fine-tune Diffusion Models without Any Reward Modelcited by 246
  9. 2024From r to Q^*: Your Language Model is Secretly a Q-Functioncited by 242
  10. 2023Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewardscited by 232
  11. 2024VideoPhy: Evaluating Physical Commonsense for Video Generationcited by 215
  12. 2025DiffusionNFT: Online Diffusion Reinforcement with Forward Processcited by 204
  13. 2023StyleDrop: Text-to-Image Generation in Any Stylecited by 197
  14. 2025Unified Reward Model for Multimodal Understanding and Generationcited by 194
  15. 2023HIVE: Harnessing Human Feedback for Instructional Visual Editingcited by 172
  16. 2024RM-Bench: Benchmarking Reward Models of Language Models with Subtlety and Stylecited by 168
  17. 2025MixGRPO: Unlocking Flow-based GRPO Efficiency with Mixed ODE-SDEcited by 168
  18. 2023Grounded Text-to-Image Synthesis with Attention Refocusingcited by 163
  19. 2024Fine-Tuning of Continuous-Time Diffusion Models as Entropy-Regularized Controlcited by 136
  20. 2025Alpamayo-R1: Bridging Reasoning and Action Prediction for Generalizable Autonomous Driving in the Long Tailcited by 136
  21. 2024GenAI-Bench: Evaluating and Improving Compositional Text-to-Visual Generationcited by 130
  22. 2024ReNO: Enhancing One-step Text-to-Image Models through Reward-based Noise Optimizationcited by 115
  23. 2025X-Omni: Reinforcement Learning Makes Discrete Autoregressive Image Generative Models Great Againcited by 105
  24. 2024Video Diffusion Alignment via Reward Gradientscited by 101
  25. 2024Aligning Diffusion Models by Optimizing Human Utilitycited by 99
  26. 2023Is RLHF More Difficult than Standard RL?cited by 96
  27. 2024Understanding Reinforcement Learning-Based Fine-Tuning of Diffusion Models: A Tutorial and Reviewcited by 92
  28. 2023A Survey of Safety and Trustworthiness of Large Language Models through the Lens of Verification and Validationcited by 91
  29. 2025Test-time Alignment of Diffusion Models without Reward Over-optimizationcited by 89
  30. 2024Improving Text-to-Image Consistency via Automatic Prompt Optimizationcited by 88
  31. 2025Flow Matching Policy Gradientscited by 86
  32. 2023LLMScore: Unveiling the Power of Large Language Models in Text-to-Image Synthesis Evaluationcited by 85
  33. 2023InstructVideo: Instructing Video Diffusion Models with Human Feedbackcited by 85
  34. 2024Self-Play Fine-Tuning of Diffusion Models for Text-to-Image Generationcited by 85
  35. 2025SimpleAR: Pushing the Frontier of Autoregressive Visual Generation through Pretraining, SFT, and RLcited by 85
  36. 2024Aesthetic Post-Training Diffusion Models from Generic Preferences with Step-by-step Preference Optimizationcited by 81
  37. 2024Large-scale Reinforcement Learning for Diffusion Modelscited by 78
  38. 2023Contrastive Preference Learning: Learning from Human Feedback without RLcited by 77

50 most cited of 418 All 418 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Rich Human Feedback for Text-to-Image Generation
  2. 2024Bridging the Gap: Aligning Text-to-Image Diffusion Models with Specific Feedback
  3. 2024Enhancing Image Caption Generation Using Reinforcement Learning with Human Feedback
  4. 2025M3T2IBench: A Large-Scale Multi-Category, Multi-Instance, Multi-Relation Text-to-Image Benchmark
  5. 2025Enhancing Reward Models for High-quality Image Generation: Beyond Text-Image Alignment
  6. 2025InstructEngine: Instruction-driven Text-to-Image Alignment
  7. 2024EvalMuse-40K: A Reliable and Fine-Grained Benchmark with Comprehensive Human Annotations for Text-to-Image Generation Model Evaluation
  8. 2024Precision or Recall? An Analysis of Image Captions for Training Text-to-Image Generation Model