# Aesthetic Post-Training Diffusion Models from Generic Preferences with Step-by-step Preference Optimization

arXiv: https://arxiv.org/abs/2406.04314 · HTML: https://mlsanity.com/paper/2406.04314/

## Key facts

- Cited by 84 papers in the map (data through 14 Oct 2026).
- Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models.
- Builds on 28 of its 35 references that are in the map.
- First posted on arXiv 6 Jun 2024.

## Placed in

- Diffusion models / [Diffusion and flow policies for robotics](https://mlsanity.com/diffusion/diffusion-and-flow-policies-for-robotics/)
- LLM alignment / RLHF / preference optimisation / [Diffusion and multimodal model alignment](https://mlsanity.com/alignment/diffusion-and-multimodal-model-alignment/)

## Builds on (most cited first, 28 of 28)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis](https://mlsanity.com/paper/2307.01952/) (2023)
- [Scaling Rectified Flow Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2403.03206/) (2024)
- [Scaling Autoregressive Models for Content-Rich Text-to-Image Generation](https://mlsanity.com/paper/2206.10789/) (2022)
- [Training Diffusion Models with Reinforcement Learning](https://mlsanity.com/paper/2305.13301/) (2023)
- [Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generation](https://mlsanity.com/paper/2305.01569/) (2023)
- [GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignment](https://mlsanity.com/paper/2310.11513/) (2023)
- [Human Preference Score v2: A Solid Benchmark for Evaluating Human Preferences of Text-to-Image Synthesis](https://mlsanity.com/paper/2306.09341/) (2023)
- [Diffusion Model Alignment Using Direct Preference Optimization](https://mlsanity.com/paper/2311.12908/) (2023)
- [Directly Fine-Tuning Diffusion Models on Differentiable Rewards](https://mlsanity.com/paper/2309.17400/) (2023)
- [Aligning Text-to-Image Models using Human Feedback](https://mlsanity.com/paper/2302.12192/) (2023)
- [Aligning Text-to-Image Diffusion Models with Reward Backpropagation](https://mlsanity.com/paper/2310.03739/) (2023)
- [Using Human Feedback to Fine-tune Diffusion Models without Any Reward Model](https://mlsanity.com/paper/2311.13231/) (2023)
- [Glyph-ByT5: A Customized Text Encoder for Accurate Visual Text Rendering](https://mlsanity.com/paper/2403.09622/) (2024)
- [Large-scale Reinforcement Learning for Diffusion Models](https://mlsanity.com/paper/2401.12244/) (2024)
- [MAPO: Advancing Multilingual Reasoning through Multilingual Alignment-as-Preference Optimization](https://mlsanity.com/paper/2401.06838/) (2024)
- [A Dense Reward View on Aligning Text-to-Image Diffusion with Preference](https://mlsanity.com/paper/2402.08265/) (2024)
- [Enhancing Diffusion Models with Text-Encoder Reinforcement Learning](https://mlsanity.com/paper/2311.15657/) (2023)

## Built on by (most cited first, 30 of 84)

- [Flow-GRPO: Training Flow Matching Models via Online RL](https://mlsanity.com/paper/2505.05470/) (2025)
- [DanceGRPO: Unleashing GRPO on Visual Generation](https://mlsanity.com/paper/2505.07818/) (2025)
- [MixGRPO: Unlocking Flow-based GRPO Efficiency with Mixed ODE-SDE](https://mlsanity.com/paper/2507.21802/) (2025)
- [TempFlow-GRPO: When Timing Matters for GRPO in Flow Models](https://mlsanity.com/paper/2508.04324/) (2025)
- [BranchGRPO: Stable and Efficient GRPO with Structured Branching in Diffusion Models](https://mlsanity.com/paper/2509.06040/) (2025)
- [Glyph-ByT5-v2: A Strong Aesthetic Baseline for Accurate Multilingual Visual Text Rendering](https://mlsanity.com/paper/2406.10208/) (2024)
- [Diffusion Model as a Noise-Aware Latent Reward Model for Step-Level Preference Optimization](https://mlsanity.com/paper/2502.01051/) (2025)
- [DenseDPO: Fine-Grained Temporal Preference Optimization for Video Diffusion Models](https://mlsanity.com/paper/2506.03517/) (2025)
- [IterComp: Iterative Composition-Aware Feedback Learning from Model Gallery for Text-to-Image Generation](https://mlsanity.com/paper/2410.07171/) (2024)
- [Directly Aligning the Full Diffusion Trajectory with Fine-Grained Human Preference](https://mlsanity.com/paper/2509.06942/) (2025)
- [Tuning Timestep-Distilled Diffusion Model Using Pairwise Sample Optimization](https://mlsanity.com/paper/2410.03190/) (2024)
- [Fine-Grained GRPO for Precise Preference Alignment in Flow Models](https://mlsanity.com/paper/2510.01982/) (2025)
- [Flash-DMD: Towards High-Fidelity Few-Step Image Generation with Efficient Distillation and Joint Reinforcement Learning](https://mlsanity.com/paper/2511.20549/) (2025)
- [Generalizing Alignment Paradigm of Text-to-Image Generation with Preferences through f-divergence Minimization](https://mlsanity.com/paper/2409.09774/) (2024)
- [Principled RL for Flow Matching Emerges from the Chunk-level Policy Optimization](https://mlsanity.com/paper/2510.21583/) (2025)
- [ReAlign: Text-to-Motion Generation via Step-Aware Reward-Guided Alignment](https://mlsanity.com/paper/2511.19217/) (2025)
- [UniGRPO: Unified Policy Optimization for Reasoning-Driven Visual Generation](https://mlsanity.com/paper/2603.23500/) (2026)
- [Aligning Few-Step Diffusion Models with Dense Reward Difference Learning](https://mlsanity.com/paper/2411.11727/) (2024)
- [Inference-Time Alignment Control for Diffusion Models with Reinforcement Learning Guidance](https://mlsanity.com/paper/2508.21016/) (2025)
- [TAGRPO: Boosting GRPO on Image-to-Video Generation with Direct Trajectory Alignment](https://mlsanity.com/paper/2601.05729/) (2026)
- [Alleviating Sparse Rewards by Modeling Step-Wise and Long-Term Sampling Effects in Flow-Based GRPO](https://mlsanity.com/paper/2602.06422/) (2026)
- [ImageReFL: Balancing Quality and Diversity in Human-Aligned Diffusion Models](https://mlsanity.com/paper/2505.22569/) (2025)
- [Towards Self-Improvement of Diffusion Models via Group Preference Optimization](https://mlsanity.com/paper/2505.11070/) (2025)
- [PaCo-RL: Advancing Reinforcement Learning for Consistent Image Generation with Pairwise Reward Modeling](https://mlsanity.com/paper/2512.04784/) (2025)
- [VisualPrompter: Semantic-Aware Prompt Optimization with Visual Feedback for Text-to-Image Synthesis](https://mlsanity.com/paper/2506.23138/) (2025)
- [Inversion-DPO: Precise and Efficient Post-Training for Diffusion Models](https://mlsanity.com/paper/2507.11554/) (2025)
- [UltraFlux: Data-Model Co-Design for High-quality Native 4K Text-to-Image Generation across Diverse Aspect Ratios](https://mlsanity.com/paper/2511.18050/) (2025)
- [Co-GRPO: Co-Optimized Group Relative Policy Optimization for Masked Diffusion Model](https://mlsanity.com/paper/2512.22288/) (2025)
- [MoCha:End-to-End Video Character Replacement without Structural Guidance](https://mlsanity.com/paper/2601.08587/) (2026)
- [LeapAlign: Post-Training Flow Matching Models at Any Generation Step by Building Two-Step Trajectories](https://mlsanity.com/paper/2604.15311/) (2026)

## Abstract (arXiv)

Generating visually appealing images is fundamental to modern text-to-image generation models. A potential solution to better aesthetics is direct preference optimization (DPO), which has been applied to diffusion models to improve general image quality including prompt alignment and aesthetics. Popular DPO methods propagate preference labels from clean image pairs to all the intermediate steps along the two generation trajectories. However, preference labels provided in existing datasets are blended with layout and aesthetic opinions, which would disagree with aesthetic preference. Even if aesthetic labels were provided (at substantial cost), it would be hard for the two-trajectory methods to capture nuanced visual differences at different steps. To improve aesthetics economically, this paper uses existing generic preference data and introduces step-by-step preference optimization (SPO) that discards the propagation strategy and allows fine-grained image details to be assessed. Specifically, at each denoising step, we 1) sample a pool of candidates by denoising from a shared noise latent, 2) use a step-aware preference model to find a suitable win-lose pair to supervise the diffusion model, and 3) randomly select one from the pool to initialize the next denoising step. This strategy ensures that diffusion models focus on the subtle, fine-grained visual differences instead of layout aspect. We find that aesthetics can be significantly enhanced by accumulating these improved minor differences. When fine-tuning Stable Diffusion v1.5 and SDXL, SPO yields significant improvements in aesthetics compared with existing DPO methods while not sacrificing image-text alignment compared with vanilla models. Moreover, SPO converges much faster than DPO methods due to the use of more correct preference labels provided by the step-aware preference model.