# Reinforced Preference Optimization for Reasoning-Augmented Recommendations

arXiv: https://arxiv.org/abs/2605.21967 · HTML: https://mlsanity.com/paper/2605.21967/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in LLM reasoning for recommendation, one of 17 directions in LLM reasoning.
- Builds on 14 of its 38 references that are in the map.
- First posted on arXiv 21 May 2026.

## Placed in

- LLM reasoning / [LLM reasoning for recommendation](https://mlsanity.com/lr/llm-reasoning-for-recommendation/)

## Builds on (most cited first, 14 of 14)

- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [A Survey of Large Language Models](https://mlsanity.com/paper/2303.18223/) (2023)
- [Session-based Recommendations with Recurrent Neural Networks](https://mlsanity.com/paper/1511.06939/) (2015)
- [Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning](https://mlsanity.com/paper/2506.01939/) (2025)
- [Large Language Models: A Survey](https://mlsanity.com/paper/2402.06196/) (2024)
- [From LLM Reasoning to Autonomous AI Agents: A Comprehensive Review](https://mlsanity.com/paper/2504.19678/) (2025)
- [Can ChatGPT Defend its Belief in Truth? Evaluating LLM Reasoning via Debate](https://mlsanity.com/paper/2305.13160/) (2023)
- [Reinforced Latent Reasoning for LLM-based Recommendation](https://mlsanity.com/paper/2505.19092/) (2025)
- [Decoding Matters: Addressing Amplification Bias and Homogeneity Issue for LLM-based Recommendation](https://mlsanity.com/paper/2406.14900/) (2024)
- [Robust Answers, Fragile Logic: Probing the Decoupling Hypothesis in LLM Reasoning](https://mlsanity.com/paper/2505.17406/) (2025)
- [Negative Sampling in Recommendation: A Survey and Future Directions](https://mlsanity.com/paper/2409.07237/) (2024)
- [Reinforced Preference Optimization for Recommendation](https://mlsanity.com/paper/2510.12211/) (2025)

## Built on by (most cited first, 1 of 1)

- [Towards Efficient Reasoning in LLM-Based Recommender Systems via Model Merging](https://mlsanity.com/paper/2608.10447/) (2026)

## Abstract (arXiv)

Recommender systems are critical for delivering personalized content across digital platforms, and recent advances in Large Language Models (LLMs) offer new opportunities to enhance them with richer world knowledge and explicit reasoning capabilities. With the help of reasoning knowledge, recommendations can better infer users' underlying intents, adapt to evolving preferences, and leverage semantic relationships for improved accuracy and interpretability. However, existing reasoning-based recommendation methods often fail to fully align the LLM's reasoning process with recommendation-specific objectives due to structural disruption during integration and difficulties in translating free-form generation into accurate item predictions. In this paper, we introduce RPORec, a reinforced preference optimization framework that unifies an LLM backbone's reasoning ability with a dedicated recommendation head (Rechead) for precise item retrieval. RPORec comprises two stages: (1) Reasoning-Augmented Recommendation Modeling, where high-quality Chain-of-Thought (CoT) reasoning is generated and used as auxiliary knowledge to guide the Rechead in learning recommendation-specific representations; and (2) Advanced Reasoning Refinement and Alignment, in which the trained Rechead produces verifiable rewards to fine-tune the LLM backbone via reinforcement learning, enhancing reasoning quality, structural consistency, and task relevance. Extensive experiments on public benchmarks and large-scale online deployments show that RPORec consistently outperforms state-of-the-art LLM-based recommendation methods, demonstrating the effectiveness of reasoning-augmented recommendation modeling in real-world systems.