# Human Preference Score: Better Aligning Text-to-Image Models with Human Preference

arXiv: https://arxiv.org/abs/2303.14420 · HTML: https://mlsanity.com/paper/2303.14420/

## Key facts

- Cited by 287 papers in the map (data through 14 Oct 2026).
- Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models.
- Builds on 40 of its 48 references that are in the map.
- First posted on arXiv 25 Mar 2023.

## Placed in

- Diffusion models / [Diffusion and flow policies for robotics](https://mlsanity.com/diffusion/diffusion-and-flow-policies-for-robotics/)
- LLM alignment / RLHF / preference optimisation / [Diffusion and multimodal model alignment](https://mlsanity.com/alignment/diffusion-and-multimodal-model-alignment/)

## Builds on (most cited first, 30 of 40)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [Generative Modeling by Estimating Gradients of the Data Distribution](https://mlsanity.com/paper/1907.05600/) (2019)
- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models](https://mlsanity.com/paper/2112.10741/) (2021)
- [DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation](https://mlsanity.com/paper/2208.12242/) (2022)
- [Improved Techniques for Training GANs](https://mlsanity.com/paper/1606.03498/) (2016)
- [InstructPix2Pix: Learning to Follow Image Editing Instructions](https://mlsanity.com/paper/2211.09800/) (2022)
- [RAFT: Recurrent All-Pairs Field Transforms for Optical Flow](https://mlsanity.com/paper/2003.12039/) (2020)
- [Microsoft COCO Captions: Data Collection and Evaluation Server](https://mlsanity.com/paper/1504.00325/) (2015)
- [ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness](https://mlsanity.com/paper/1811.12231/) (2018)
- [FlowNet: Learning Optical Flow with Convolutional Networks](https://mlsanity.com/paper/1504.06852/) (2015)
- [A Large Dataset to Train Convolutional Networks for Disparity, Optical Flow, and Scene Flow Estimation](https://mlsanity.com/paper/1512.02134/) (2015)
- [Pseudo Numerical Methods for Diffusion Models on Manifolds](https://mlsanity.com/paper/2202.09778/) (2022)
- [Aligning Text-to-Image Models using Human Feedback](https://mlsanity.com/paper/2302.12192/) (2023)
- [ELITE: Encoding Visual Concepts into Textual Embeddings for Customized Text-to-Image Generation](https://mlsanity.com/paper/2302.13848/) (2023)
- [Training-Free Structured Diffusion Guidance for Compositional Text-to-Image Synthesis](https://mlsanity.com/paper/2212.05032/) (2022)
- [DiffusionDB: A Large-scale Prompt Gallery Dataset for Text-to-Image Generative Models](https://mlsanity.com/paper/2210.14896/) (2022)

## Built on by (most cited first, 30 of 287)

- [RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment](https://mlsanity.com/paper/2304.06767/) (2023)
- [DanceGRPO: Unleashing GRPO on Visual Generation](https://mlsanity.com/paper/2505.07818/) (2025)
- [Aligning Text-to-Image Diffusion Models with Reward Backpropagation](https://mlsanity.com/paper/2310.03739/) (2023)
- [Improving Video Generation with Human Feedback](https://mlsanity.com/paper/2501.13918/) (2025)
- [AnyEdit: Mastering Unified High-Quality Image Editing for Any Idea](https://mlsanity.com/paper/2411.15738/) (2024)
- [DiffusionNFT: Online Diffusion Reinforcement with Forward Process](https://mlsanity.com/paper/2509.16117/) (2025)
- [HPSv3: Towards Wide-Spectrum Human Preference Score](https://mlsanity.com/paper/2508.03789/) (2025)
- [Sociotechnical Safety Evaluation of Generative AI Systems](https://mlsanity.com/paper/2310.11986/) (2023)
- [AGIQA-3K: An Open Database for AI-Generated Image Quality Assessment](https://mlsanity.com/paper/2306.04717/) (2023)
- [Rich Human Feedback for Text-to-Image Generation](https://mlsanity.com/paper/2312.10240/) (2023)
- [WorldModelBench: Judging Video Generation Models As World Models](https://mlsanity.com/paper/2502.20694/) (2025)
- [Aligning Diffusion Models by Optimizing Human Utility](https://mlsanity.com/paper/2404.04465/) (2024)
- [Envisioning Beyond the Pixels: Benchmarking Reasoning-Informed Visual Editing](https://mlsanity.com/paper/2504.02826/) (2025)
- [Test-time Alignment of Diffusion Models without Reward Over-optimization](https://mlsanity.com/paper/2501.05803/) (2025)
- [T2V-Turbo: Breaking the Quality Bottleneck of Video Consistency Model with Mixed Reward Feedback](https://mlsanity.com/paper/2405.18750/) (2024)
- [InstructVideo: Instructing Video Diffusion Models with Human Feedback](https://mlsanity.com/paper/2312.12490/) (2023)
- [Large-scale Reinforcement Learning for Diffusion Models](https://mlsanity.com/paper/2401.12244/) (2024)
- [SVGDreamer: Text Guided SVG Generation with Diffusion Model](https://mlsanity.com/paper/2312.16476/) (2023)
- [Strengthening Multimodal Large Language Model with Bootstrapped Preference Optimization](https://mlsanity.com/paper/2403.08730/) (2024)
- [OmniSVG: A Unified Scalable Vector Graphics Generation Model](https://mlsanity.com/paper/2504.06263/) (2025)
- [RewardDance: Reward Scaling in Visual Generation](https://mlsanity.com/paper/2509.08826/) (2025)
- [Empowering LLMs to Understand and Generate Complex Vector Graphics](https://mlsanity.com/paper/2412.11102/) (2024)
- [EditScore: Unlocking Online RL for Image Editing via High-Fidelity Reward Modeling](https://mlsanity.com/paper/2509.23909/) (2025)
- [EditReward: A Human-Aligned Reward Model for Instruction-Guided Image Editing](https://mlsanity.com/paper/2509.26346/) (2025)
- [On the Design Fundamentals of Diffusion Models: A Survey](https://mlsanity.com/paper/2306.04542/) (2023)
- [Deep Reward Supervisions for Tuning Text-to-Image Diffusion Models](https://mlsanity.com/paper/2405.00760/) (2024)
- [A Dense Reward View on Aligning Text-to-Image Diffusion with Preference](https://mlsanity.com/paper/2402.08265/) (2024)
- [T2V-Turbo-v2: Enhancing Video Generation Model Post-Training through Data, Reward, and Conditional Guidance Design](https://mlsanity.com/paper/2410.05677/) (2024)
- [BLIP3o-NEXT: Next Frontier of Native Image Generation](https://mlsanity.com/paper/2510.15857/) (2025)
- [Distribution Matching Distillation Meets Reinforcement Learning](https://mlsanity.com/paper/2511.13649/) (2025)

## Abstract (arXiv)

Recent years have witnessed a rapid growth of deep generative models, with text-to-image models gaining significant attention from the public. However, existing models often generate images that do not align well with human preferences, such as awkward combinations of limbs and facial expressions. To address this issue, we collect a dataset of human choices on generated images from the Stable Foundation Discord channel. Our experiments demonstrate that current evaluation metrics for generative models do not correlate well with human choices. Thus, we train a human preference classifier with the collected dataset and derive a Human Preference Score (HPS) based on the classifier. Using HPS, we propose a simple yet effective method to adapt Stable Diffusion to better align with human preferences. Our experiments show that HPS outperforms CLIP in predicting human choices and has good generalization capability toward images generated from other models. By tuning Stable Diffusion with the guidance of HPS, the adapted model is able to generate images that are more preferred by human users. The project page is available here: https://tgxs002.github.io/align_sd_web/ .