# MTRec: Learning to Align with User Preferences via Mental Reward Models

arXiv: https://arxiv.org/abs/2509.22807 · HTML: https://mlsanity.com/paper/2509.22807/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Preference alignment for recommenders and RAG, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 26 of its 41 references that are in the map.
- First posted on arXiv 26 Sep 2025.

## Placed in

- LLM alignment / RLHF / preference optimisation / [Preference alignment for recommenders and RAG](https://mlsanity.com/alignment/preference-alignment-for-recommenders-and-rag/)

## Builds on (most cited first, 26 of 26)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Generative Adversarial Imitation Learning](https://mlsanity.com/paper/1606.03476/) (2016)
- [Self-Attentive Sequential Recommendation](https://mlsanity.com/paper/1808.09781/) (2018)
- [Wide & Deep Learning for Recommender Systems](https://mlsanity.com/paper/1606.07792/) (2016)
- [Image-based Recommendations on Styles and Substitutes](https://mlsanity.com/paper/1506.04757/) (2015)
- [DeepFM: A Factorization-Machine based Neural Network for CTR Prediction](https://mlsanity.com/paper/1703.04247/) (2017)
- [Deep Interest Network for Click-Through Rate Prediction](https://mlsanity.com/paper/1706.06978/) (2017)
- [Distributional Reinforcement Learning with Quantile Regression](https://mlsanity.com/paper/1710.10044/) (2017)
- [Deep & Cross Network for Ad Click Predictions](https://mlsanity.com/paper/1708.05123/) (2017)
- [Deep Interest Evolution Network for Click-Through Rate Prediction](https://mlsanity.com/paper/1809.03672/) (2018)
- [A Survey of Inverse Reinforcement Learning: Challenges, Methods and Progress](https://mlsanity.com/paper/1806.06877/) (2018)
- [Bias and Debias in Recommender System: A Survey and Future Directions](https://mlsanity.com/paper/2010.03240/) (2020)
- [IQ-Learn: Inverse soft-Q Learning for Imitation](https://mlsanity.com/paper/2106.12142/) (2021)
- [Top-K Off-Policy Correction for a REINFORCE Recommender System](https://mlsanity.com/paper/1812.02353/) (2018)
- [Product-based Neural Networks for User Response Prediction](https://mlsanity.com/paper/1611.00144/) (2016)
- [Offline A/B testing for Recommender Systems](https://mlsanity.com/paper/1801.07030/) (2018)
- [Virtual-Taobao: Virtualizing Real-world Online Retail Environment for Reinforcement Learning](https://mlsanity.com/paper/1805.10000/) (2018)
- [Denoising Implicit Feedback for Recommendation](https://mlsanity.com/paper/2006.04153/) (2020)
- [LinRec: Linear Attention Mechanism for Long-term Sequential Recommender Systems](https://mlsanity.com/paper/2411.01537/) (2024)
- [Sampler Design for Implicit Feedback Data by Noisy-label Robust Learning](https://mlsanity.com/paper/2007.07204/) (2020)
- [Capturing Delayed Feedback in Conversion Rate Prediction via Elapsed-Time Sampling](https://mlsanity.com/paper/2012.03245/) (2020)
- [An Adversarial Imitation Click Model for Information Retrieval](https://mlsanity.com/paper/2104.06077/) (2021)
- [Generative Inverse Deep Reinforcement Learning for Online Recommendation](https://mlsanity.com/paper/2011.02248/) (2020)
- [Unbiased Pairwise Learning from Implicit Feedback for Recommender Systems without Biased Variance Control](https://mlsanity.com/paper/2304.05066/) (2023)

## Built on by (most cited first, 1 of 1)

- [Inverse Theory of Mind Modeling for Content Recommendation: From Web Browsing to Dynamic Intelligent Interfaces](https://mlsanity.com/paper/2608.11354/) (2026)

## Abstract (arXiv)

Recommendation models are predominantly trained using implicit user feedback, since explicit feedback is often costly to obtain. However, implicit feedback, such as clicks, does not always reflect users' real preferences. For example, a user might click on a news article because of its attractive headline, but end up feeling uncomfortable after reading the content. In the absence of explicit feedback, such erroneous implicit signals may severely mislead recommender systems. In this paper, we propose MTRec, a novel sequential recommendation framework designed to align with real user preferences by uncovering their internal satisfaction on recommended items. Specifically, we introduce a mental reward model to quantify user satisfaction and propose a distributional inverse reinforcement learning approach to learn it. The learned mental reward model is then used to guide recommendation models to better align with users' real preferences. Our experiments show that MTRec brings significant improvements to a variety of recommendation models. We also deploy MTRec on an industrial short video platform and observe a 7 percent increase in average user viewing time.