# ImageReFL: Balancing Quality and Diversity in Human-Aligned Diffusion Models

arXiv: https://arxiv.org/abs/2505.22569 · HTML: https://mlsanity.com/paper/2505.22569/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models.
- Builds on 23 of its 25 references that are in the map.
- First posted on arXiv 28 May 2025.

## Placed in

- Diffusion models / [Diffusion and flow policies for robotics](https://mlsanity.com/diffusion/diffusion-and-flow-policies-for-robotics/)
- LLM alignment / RLHF / preference optimisation / [Diffusion and multimodal model alignment](https://mlsanity.com/alignment/diffusion-and-multimodal-model-alignment/)

## Builds on (most cited first, 23 of 23)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detection](https://mlsanity.com/paper/2203.03605/) (2022)
- [ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generation](https://mlsanity.com/paper/2304.05977/) (2023)
- [Training Diffusion Models with Reinforcement Learning](https://mlsanity.com/paper/2305.13301/) (2023)
- [Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generation](https://mlsanity.com/paper/2305.01569/) (2023)
- [Human Preference Score v2: A Solid Benchmark for Evaluating Human Preferences of Text-to-Image Synthesis](https://mlsanity.com/paper/2306.09341/) (2023)
- [Diffusion Model Alignment Using Direct Preference Optimization](https://mlsanity.com/paper/2311.12908/) (2023)
- [DPOK: Reinforcement Learning for Fine-tuning Text-to-Image Diffusion Models](https://mlsanity.com/paper/2305.16381/) (2023)
- [Perception Prioritized Training of Diffusion Models](https://mlsanity.com/paper/2204.00227/) (2022)
- [Aligning Text-to-Image Diffusion Models with Reward Backpropagation](https://mlsanity.com/paper/2310.03739/) (2023)
- [Learning Multi-dimensional Human Preference for Text-to-Image Generation](https://mlsanity.com/paper/2405.14705/) (2024)
- [Aesthetic Post-Training Diffusion Models from Generic Preferences with Step-by-step Preference Optimization](https://mlsanity.com/paper/2406.04314/) (2024)
- [Deep Reward Supervisions for Tuning Text-to-Image Diffusion Models](https://mlsanity.com/paper/2405.00760/) (2024)
- [On Analyzing Generative and Denoising Capabilities of Diffusion-based Deep Generative Models](https://mlsanity.com/paper/2206.00070/) (2022)
- [Elucidating Optimal Reward-Diversity Tradeoffs in Text-to-Image Diffusion Models](https://mlsanity.com/paper/2409.06493/) (2024)
- [StyleDomain: Efficient and Lightweight Parameterizations of StyleGAN for One-shot and Few-shot Domain Adaptation](https://mlsanity.com/paper/2212.10229/) (2022)

## Built on by (most cited first, 8 of 8)

- [Understanding Sampler Stochasticity in Training Diffusion Models for RLHF](https://mlsanity.com/paper/2510.10767/) (2025)
- [LeapAlign: Post-Training Flow Matching Models at Any Generation Step by Building Two-Step Trajectories](https://mlsanity.com/paper/2604.15311/) (2026)
- [Beyond the Dirac Delta: Mitigating Diversity Collapse in Reinforcement Fine-Tuning for Versatile Image Generation](https://mlsanity.com/paper/2601.12401/) (2026)
- [DiReCT: Disentangled Regularization of Contrastive Trajectories for Physics-Refined Video Generation](https://mlsanity.com/paper/2603.25931/) (2026)
- [Flow Matching with Injected Noise for Offline-to-Online Reinforcement Learning](https://mlsanity.com/paper/2602.18117/) (2026)
- [Policy-based Tuning of Autoregressive Image Models with Instance- and Distribution-Level Rewards](https://mlsanity.com/paper/2603.23086/) (2026)
- [Calibri: Enhancing Diffusion Transformers via Parameter-Efficient Calibration](https://mlsanity.com/paper/2603.24800/) (2026)
- [ExploreNet: Learning Where to Explore in Diffusion GRPO](https://mlsanity.com/paper/2609.38329/) (2026)

## Abstract (arXiv)

Recent advances in diffusion models have led to impressive image generation capabilities, but aligning these models with human preferences remains challenging. Reward-based fine-tuning using models trained on human feedback improves alignment but often harms diversity, producing less varied outputs. In this work, we address this trade-off with two contributions. First, we introduce textitcombined generation, a novel sampling strategy that applies a reward-tuned diffusion model only in the later stages of the generation process, while preserving the base model for earlier steps. This approach mitigates early-stage overfitting and helps retain global structure and diversity. Second, we propose textitImageReFL, a fine-tuning method that improves image diversity with minimal loss in quality by training on real images and incorporating multiple regularizers, including diffusion and ReFL losses. Our approach outperforms conventional reward tuning methods on standard quality and diversity metrics. A user study further confirms that our method better balances human preference alignment and visual diversity. The source code can be found at https://github.com/ControlGenAI/ImageReFL .