mlsanityData to 14 Oct 2026

MTRec: Learning to Align with User Preferences via Mental Reward Models

arXiv 2509.22807 ↗First posted 26 Sep 2025cs.IR cs.AIcited by 1

On the map: LLM alignment / RLHF / preference optimisation → Preference alignment for recommenders and RAG

Abstract (arXiv)

Recommendation models are predominantly trained using implicit user feedback, since explicit feedback is often costly to obtain. However, implicit feedback, such as clicks, does not always reflect users' real preferences. For example, a user might click on a news article because of its attractive headline, but end up feeling uncomfortable after reading the content. In the absence of explicit feedback, such erroneous implicit signals may severely mislead recommender systems. In this paper, we propose MTRec, a novel sequential recommendation framework designed to align with real user preferences by uncovering their internal satisfaction on recommended items. Specifically, we introduce a mental reward model to quantify user satisfaction and propose a distributional inverse reinforcement learning approach to learn it. The learned mental reward model is then used to guide recommendation models to better align with users' real preferences. Our experiments show that MTRec brings significant improvements to a variety of recommendation models. We also deploy MTRec on an industrial short video platform and observe a 7 percent increase in average user viewing time.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Preference alignment for recommenders and RAG, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 26 of its 41 references that are in the map. First posted on arXiv 26 Sep 2025.

Builds on

26 of 41 references.

References · 26

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  4. 2016Generative Adversarial Imitation Learningcited by 1,597
  5. 2018Self-Attentive Sequential Recommendationcited by 1,184
  6. 2016Wide & Deep Learning for Recommender Systemscited by 747
  7. 2015Image-based Recommendations on Styles and Substitutescited by 706
  8. 2017DeepFM: A Factorization-Machine based Neural Network for CTR Predictioncited by 676
  9. 2017Deep Interest Network for Click-Through Rate Predictioncited by 544
  10. 2017Distributional Reinforcement Learning with Quantile Regressioncited by 474
  11. 2017Deep & Cross Network for Ad Click Predictionscited by 393
  12. 2018Deep Interest Evolution Network for Click-Through Rate Predictioncited by 338
14 more
  1. 2018A Survey of Inverse Reinforcement Learning: Challenges, Methods and Progresscited by 298
  2. 2020Bias and Debias in Recommender System: A Survey and Future Directionscited by 240
  3. 2021IQ-Learn: Inverse soft-Q Learning for Imitationcited by 193
  4. 2018Top-K Off-Policy Correction for a REINFORCE Recommender Systemcited by 182
  5. 2016Product-based Neural Networks for User Response Predictioncited by 145
  6. 2018Offline A/B testing for Recommender Systemscited by 96
  7. 2018Virtual-Taobao: Virtualizing Real-world Online Retail Environment for Reinforcement Learningcited by 80
  8. 2020Denoising Implicit Feedback for Recommendationcited by 80
  9. 2024LinRec: Linear Attention Mechanism for Long-term Sequential Recommender Systemscited by 21
  10. 2020Sampler Design for Implicit Feedback Data by Noisy-label Robust Learningcited by 18
  11. 2020Capturing Delayed Feedback in Conversion Rate Prediction via Elapsed-Time Samplingcited by 18
  12. 2021An Adversarial Imitation Click Model for Information Retrievalcited by 18
  13. 2020Generative Inverse Deep Reinforcement Learning for Online Recommendationcited by 12
  14. 2023Unbiased Pairwise Learning from Implicit Feedback for Recommender Systems without Biased Variance Controlcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Inverse Theory of Mind Modeling for Content Recommendation: From Web Browsing to Dynamic Intelligent Interfacescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026FlexRec: Adapting LLM-based Recommenders for Flexible Needs via Reinforcement Learning
  2. 2023Adversarial Batch Inverse Reinforcement Learning: Learn to Reward from Imperfect Demonstration for Interactive Recommendation
  3. 2025What Makes LLMs Effective Sequential Recommenders? A Study on Preference Intensity and Temporal Context
  4. 2019Learning to Recommend from Sparse Data via Generative User Feedback ↗
  5. 2020Offline Meta-level Model-based Reinforcement Learning Approach for Cold-Start Recommendation
  6. 2025RecoMind: A Reinforcement Learning Framework for Optimizing In-Session User Satisfaction in Recommendation Systems
  7. 2026LLM-Enhanced Reinforcement Learning for Long-Term User Satisfaction in Interactive Recommendation
  8. 2022Multi-Task Fusion via Reinforcement Learning for Long-Term User Satisfaction in Recommender Systems