# Direct Preference Optimization for LLM-Enhanced Recommendation Systems

arXiv: https://arxiv.org/abs/2410.05939 · HTML: https://mlsanity.com/paper/2410.05939/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Oct 2024.

## Built on by (most cited first, 4 of 4)

- [Rec-R1: Bridging Generative Large Language Models and User-Centric Recommendation Systems via Reinforcement Learning](https://mlsanity.com/paper/2503.24289/) (2025)
- [MASS-DPO: Multi-negative Active Sample Selection for Direct Policy Optimization](https://mlsanity.com/paper/2605.10784/) (2026)
- [U-MASK: User-adaptive Spatio-Temporal Masking for Personalized Mobile AI Applications](https://mlsanity.com/paper/2601.06867/) (2026)
- [Aligning Multimodal Sequential Recommendations via Robust Direct Preference Optimization with Sparse MoE](https://mlsanity.com/paper/2603.29259/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) have exhibited remarkable performance across a wide range of domains, motivating research into their potential for recommendation systems. Early efforts have leveraged LLMs' rich knowledge and strong generalization capabilities via in-context learning, where recommendation tasks are framed as prompts. However, LLM performance in recommendation scenarios remains limited due to the mismatch between their pretraining objectives and recommendation tasks, as well as the lack of recommendation-specific data during pretraining. To address these challenges, we propose DPO4Rec, a novel framework that integrates Direct Preference Optimization (DPO) into LLM-enhanced recommendation systems. First, we prompt the LLM to infer user preferences from historical interactions, which are then used to augment traditional ID-based sequential recommendation models. Next, we train a reward model based on knowledge-augmented recommendation architectures to assess the quality of LLM-generated reasoning. Using this, we select the highest- and lowest-ranked responses from N samples to construct a dataset for LLM fine-tuning. Finally, we apply a structure alignment strategy via DPO to align the LLM's outputs with desirable recommendation behavior. Extensive experiments show that DPO4Rec significantly improves re-ranking performance over strong baselines, demonstrating enhanced instruction-following capabilities of LLMs in recommendation tasks.