# Reward-Instruct: A Reward-Centric Approach to Fast Photo-Realistic Image Generation

arXiv: https://arxiv.org/abs/2503.13070 · HTML: https://mlsanity.com/paper/2503.13070/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models.
- Builds on 63 of its 70 references that are in the map.
- First posted on arXiv 17 Mar 2025.

## Placed in

- Diffusion models / [Diffusion and flow policies for robotics](https://mlsanity.com/diffusion/diffusion-and-flow-policies-for-robotics/)
- LLM alignment / RLHF / preference optimisation / [Diffusion and multimodal model alignment](https://mlsanity.com/alignment/diffusion-and-multimodal-model-alignment/)

## Builds on (most cited first, 30 of 63)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis](https://mlsanity.com/paper/2307.01952/) (2023)
- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2204.05862/) (2022)
- [DreamFusion: Text-to-3D using 2D Diffusion](https://mlsanity.com/paper/2209.14988/) (2022)
- [CLIPScore: A Reference-free Evaluation Metric for Image Captioning](https://mlsanity.com/paper/2104.08718/) (2021)
- [Fine-Tuning Language Models from Human Preferences](https://mlsanity.com/paper/1909.08593/) (2019)
- [DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps](https://mlsanity.com/paper/2206.00927/) (2022)
- [Consistency Models](https://mlsanity.com/paper/2303.01469/) (2023)
- [Imagen Video: High Definition Video Generation with Diffusion Models](https://mlsanity.com/paper/2210.02303/) (2022)
- [Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time](https://mlsanity.com/paper/2203.05482/) (2022)
- [ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generation](https://mlsanity.com/paper/2304.05977/) (2023)
- [One-step Diffusion with Distribution Matching Distillation](https://mlsanity.com/paper/2311.18828/) (2023)
- [Training Diffusion Models with Reinforcement Learning](https://mlsanity.com/paper/2305.13301/) (2023)
- [Human Preference Score v2: A Solid Benchmark for Evaluating Human Preferences of Text-to-Image Synthesis](https://mlsanity.com/paper/2306.09341/) (2023)
- [Diffusion Model Alignment Using Direct Preference Optimization](https://mlsanity.com/paper/2311.12908/) (2023)
- [Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learning](https://mlsanity.com/paper/1910.00177/) (2019)
- [Adversarial Diffusion Distillation](https://mlsanity.com/paper/2311.17042/) (2023)
- [Fast Sampling of Diffusion Models with Exponential Integrator](https://mlsanity.com/paper/2204.13902/) (2022)
- [Consistency Trajectory Models: Learning Probability Flow ODE Trajectory of Diffusion](https://mlsanity.com/paper/2310.02279/) (2023)

## Built on by (most cited first, 20 of 20)

- [Advantage Weighted Matching: Aligning RL with Pretraining in Diffusion Models](https://mlsanity.com/paper/2509.25050/) (2025)
- [Ultra-Fast Language Generation via Discrete Diffusion Divergence Instruct](https://mlsanity.com/paper/2509.25035/) (2025)
- [Reinforcing Diffusion Models by Direct Group Preference Optimization](https://mlsanity.com/paper/2510.08425/) (2025)
- [TDM-R1: Reinforcing Few-Step Diffusion Models with Non-Differentiable Reward](https://mlsanity.com/paper/2603.07700/) (2026)
- [Soft-Di[M]O: Improving One-Step Discrete Image Generation with Soft Embeddings](https://mlsanity.com/paper/2509.22925/) (2025)
- [Reward-Forcing: Autoregressive Video Generation with Reward Feedback](https://mlsanity.com/paper/2601.16933/) (2026)
- [Faster and Better Alignment for Flow Matching Models via Step-aware Advantages](https://mlsanity.com/paper/2602.01591/) (2026)
- [DreamLite: A Lightweight On-Device Unified Model for Image Generation and Editing](https://mlsanity.com/paper/2603.28713/) (2026)
- [Noise Consistency Training: A Native Approach for One-Step Generator in Learning Additional Controls](https://mlsanity.com/paper/2506.19741/) (2025)
- [FP4 Explore, BF16 Train: Diffusion Reinforcement Learning via Efficient Rollout Scaling](https://mlsanity.com/paper/2604.06916/) (2026)
- [Drifting Preference Optimization for One-Step Generative Models](https://mlsanity.com/paper/2606.02521/) (2026)
- [Ultra Fast PDE Solving via Physics Guided Few-step Diffusion](https://mlsanity.com/paper/2602.03627/) (2026)
- [Preference Score Distillation: Leveraging 2D Rewards to Align Text-to-3D Generation with Human Preference](https://mlsanity.com/paper/2603.01594/) (2026)
- [Diffusion Fine-tuning with Rewarded Moment Matching Distillation](https://mlsanity.com/paper/2606.30414/) (2026)
- [OmniQuality-R: Advancing Reward Models Through All-Encompassing Quality Assessment](https://mlsanity.com/paper/2510.10609/) (2025)
- [MIRO: MultI-Reward cOnditioned pretraining improves T2I quality and efficiency](https://mlsanity.com/paper/2510.25897/) (2025)
- [Masked Auto-Regressive Variational Acceleration: Fast Inference Makes Practical Reinforcement Learning](https://mlsanity.com/paper/2511.15190/) (2025)
- [Unbiased Diffusion Variational Inversion via Principled Posterior Matching](https://mlsanity.com/paper/2605.25042/) (2026)
- [Quality-Aware Modulation for Diffusion Transformers](https://mlsanity.com/paper/2606.30934/) (2026)
- [EigDiff: Eigensample Diffusion - or how to significantly reduce training dataset size with marginal performance variation?](https://mlsanity.com/paper/2610.20999/) (2026)

## Abstract (arXiv)

This paper addresses the challenge of achieving high-quality and fast image generation that aligns with complex human preferences. While recent advancements in diffusion models and distillation have enabled rapid generation, the effective integration of reward feedback for improved abilities like controllability and preference alignment remains a key open problem. Existing reward-guided post-training approaches targeting accelerated few-step generation often deem diffusion distillation losses indispensable. However, in this paper, we identify an interesting yet fundamental paradigm shift: as conditions become more specific, well-designed reward functions emerge as the primary driving force in training strong, few-step image generative models. Motivated by this insight, we introduce Reward-Instruct, a novel and surprisingly simple reward-centric approach for converting pre-trained base diffusion models into reward-enhanced few-step generators. Unlike existing methods, Reward-Instruct does not rely on expensive yet tricky diffusion distillation losses. Instead, it iteratively updates the few-step generator's parameters by directly sampling from a reward-tilted parameter distribution. Such a training approach entirely bypasses the need for expensive diffusion distillation losses, making it favorable to scale in high image resolutions. Despite its simplicity, Reward-Instruct yields surprisingly strong performance. Our extensive experiments on text-to-image generation have demonstrated that Reward-Instruct achieves state-of-the-art results in visual quality and quantitative metrics compared to distillation-reliant methods, while also exhibiting greater robustness to the choice of reward function.