# Fine-Tuning Diffusion-Based Recommender Systems via Reinforcement Learning with Reward Function Optimization

arXiv: https://arxiv.org/abs/2511.06937 · HTML: https://mlsanity.com/paper/2511.06937/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Diffusion recommenders, one of 14 directions in Diffusion models.
- Builds on 41 of its 59 references that are in the map.
- First posted on arXiv 10 Nov 2025.

## Placed in

- Diffusion models / [Diffusion recommenders](https://mlsanity.com/diffusion/diffusion-recommenders/)
- LLM alignment / RLHF / preference optimisation / [Diffusion and multimodal model alignment](https://mlsanity.com/alignment/diffusion-and-multimodal-model-alignment/)

## Builds on (most cited first, 30 of 41)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [LightGCN: Simplifying and Powering Graph Convolution Network for Recommendation](https://mlsanity.com/paper/2002.02126/) (2020)
- [Planning with Diffusion for Flexible Behavior Synthesis](https://mlsanity.com/paper/2205.09991/) (2022)
- [Training Diffusion Models with Reinforcement Learning](https://mlsanity.com/paper/2305.13301/) (2023)
- [Graph Neural Networks for Social Recommendation](https://mlsanity.com/paper/1902.07243/) (2019)
- [Neural Graph Collaborative Filtering](https://mlsanity.com/paper/1905.08108/) (2019)
- [Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learning](https://mlsanity.com/paper/2208.06193/) (2022)
- [Aligning Text-to-Image Models using Human Feedback](https://mlsanity.com/paper/2302.12192/) (2023)
- [Variational Autoencoders for Collaborative Filtering](https://mlsanity.com/paper/1802.05814/) (2018)
- [Self-supervised Graph Learning for Recommendation](https://mlsanity.com/paper/2010.10783/) (2020)
- [Monte Carlo Gradient Estimation in Machine Learning](https://mlsanity.com/paper/1906.10652/) (2019)
- [A Survey on Generative Diffusion Model](https://mlsanity.com/paper/2209.02646/) (2022)
- [An Introduction to Deep Reinforcement Learning](https://mlsanity.com/paper/1811.12560/) (2018)
- [Efficient Diffusion Policies for Offline Reinforcement Learning](https://mlsanity.com/paper/2305.20081/) (2023)
- [Contrastive Energy Prediction for Exact Energy-Guided Diffusion Sampling in Offline Reinforcement Learning](https://mlsanity.com/paper/2304.12824/) (2023)
- [Sequential Recommender Systems: Challenges, Progress and Prospects](https://mlsanity.com/paper/2001.04830/) (2019)
- [Diffusion Model is an Effective Planner and Data Synthesizer for Multi-Task Reinforcement Learning](https://mlsanity.com/paper/2305.18459/) (2023)
- [DiffuRec: A Diffusion Model for Sequential Recommendation](https://mlsanity.com/paper/2304.00686/) (2023)
- [Diffusion Recommender Model](https://mlsanity.com/paper/2304.04971/) (2023)
- [Optimizing DDPM Sampling with Shortcut Fine-Tuning](https://mlsanity.com/paper/2301.13362/) (2023)
- [Towards Long-term Fairness in Recommendation](https://mlsanity.com/paper/2101.03584/) (2021)
- [Generate What You Prefer: Reshaping Sequential Recommendation via Guided Diffusion](https://mlsanity.com/paper/2310.20453/) (2023)
- [RecVAE: a New Variational Autoencoder for Top-N Recommendations with Implicit Feedback](https://mlsanity.com/paper/1912.11160/) (2019)
- [Exploiting Cognitive Structure for Adaptive Learning](https://mlsanity.com/paper/1905.12470/) (2019)
- [Improved Fine-Tuning by Better Leveraging Pre-Training Data](https://mlsanity.com/paper/2111.12292/) (2021)
- [Large-scale Interactive Recommendation with Tree-structured Policy Gradient](https://mlsanity.com/paper/1811.05869/) (2018)
- [Diffusion Models for Generative Outfit Recommendation](https://mlsanity.com/paper/2402.17279/) (2024)

## Abstract (arXiv)

Diffusion models recently emerged as a powerful paradigm for recommender systems, offering state-of-the-art performance by modeling the generative process of user-item interactions. However, training such models from scratch is both computationally expensive and yields diminishing returns once convergence is reached. To remedy these challenges, we propose ReFiT, a new framework that integrates Reinforcement learning (RL)-based Fine-Tuning into diffusion-based recommender systems. In contrast to prior RL approaches for diffusion models depending on external reward models, ReFiT adopts a task-aligned design: it formulates the denoising trajectory as a Markov decision process (MDP) and incorporates a collaborative signal-aware reward function that directly reflects recommendation quality. By tightly coupling the MDP structure with this reward signal, ReFiT empowers the RL agent to exploit high-order connectivity for fine-grained optimization, while avoiding the noisy or uninformative feedback common in naive reward designs. Leveraging policy gradient optimization, ReFiT maximizes exact log-likelihood of observed interactions, thereby enabling effective post hoc fine-tuning of diffusion recommenders. Comprehensive experiments on wide-ranging real-world datasets demonstrate that the proposed ReFiT framework (a) exhibits substantial performance gains over strong competitors (up to 36.3% on sequential recommendation), (b) demonstrates strong efficiency with linear complexity in the number of users or items, and (c) generalizes well across multiple diffusion-based recommendation scenarios. The source code and datasets are publicly available at https://anonymous.4open.science/r/ReFiT-4C60.